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Overview 


This report provides evidence on the implementation and early impacts of one promising effort 
to meet the needs of low-income students and local employers for skilled workers, the Valley 
Initiative for Development and Advancement (VIDA). VIDA, a community-based organization, 
is one of nine career pathways programs in the Pathways for Advancing Careers and Education 
(PACE) study sponsored by the Administration for Children and Families. 


VIDA’s primary goal is for participants to graduate with an associate’s degree or industry- 
recognized certificate in a high-demand occupation and find employment that pays a living- 
wage. VIDA supports full-time enrollment at local colleges through required attendance at 
weekly intensive counseling sessions, as well as through substantial financial assistance. For 
participants who are not college-ready, VIDA offers an accelerated basic skills academy. 


Using a rigorous research design, the study found that the VIDA program significantly increased 
the total number of college credits earned within a 24-month follow-up period. These effects 
are among the largest observed in rigorous studies of programs intended to increase low- 
income students’ enrollment and completion of post-secondary education. The program also 
increased rates of full-time college enrollment, enrollment more generally, and summer school 
enrollment. Finally, it increased the attainment of college credentials. Future reports will 
examine whether these effects translate into gains in employment and earnings. 


Primary Research Questions 


e What was the actual implementation of the intervention? Did it deviate from plans or 
expectations? 

e What were the students’ participation patterns and experiences with program services? 

e What were the main effects of VIDA on educational attainment, including college credits 
received, credentials received, and other educational outcomes? 


Purpose 


Low-income workers with only a high school education face poor and declining employment 
prospects. Postsecondary training, often at community colleges, offers one strategy for 
improving this population’s education and employment opportunities, especially if targeted to 
occupations where there is high and growing demand for skilled workers. Policymakers, 
workforce development organizations, educators, and other key stakeholders are interested in 
how to facilitate a better match between the nation’s need for a skilled workforce and the 
needs of low-income adults for employment. 


The intention of programs like VIDA is to address these issues by providing well-articulated 
training and employment steps targeted to locally in-demand jobs, combined with a range of 


Abt Associates Overview J pg. O-1 


Valley Initiative for Development and Advancement 
Implementation and Early Impact Report 
PACE 


supports. Policymakers and practitioners have shown great interest in the career pathways 
approach. But, to date, limited rigorous research is available on its effects on participants’ 
educational and economic outcomes. To assess the effectiveness of a program such as VIDA, 
the PACE evaluation uses an experimental design—that is, randomly assigning study 
participants to a “treatment” group who can access the program and a “control” 
cannot, then comparing their outcomes. 


group who 


Key Findings and Highlights 


e VIDA staff and participants described the program as a set of services, obligations and 
incentives. 


e VIDA’s direct assistance with tuition and related training expenses averaged almost $7,000 
per participant. 


e Training for nursing and allied health professions were the most common programs 
attended, followed by education, social services, and specialized trades programs. * 


e The treatment group earned significantly more college credits than the control group. 


e VIDA significantly increased rates of full-time college enrollment and enrollment more 
generally. 


e The treatment group members earned significantly more college credentials. 


Methods 


The VIDA evaluation includes an implementation study that examines the design and operation 
of the program and participation patterns of participants and an impact study that uses an 
experimental design to measure the differences in education and employment outcomes. 


Between November 2011 and September 2014, the evaluation randomly assigned 958 program 
applicants to either the treatment or the control group. Data sources were: a follow-up survey 
conducted approximately 20 months after random assignment, and administrative records 
from VIDA and college records from the local colleges that almost all VIDA participants and 
control group members attended. The evaluation also included site visits to document program 
implementation and operations. Prior to estimating VIDA impacts, the research team published 
an analysis plan specifying key hypotheses and outcome measures.” 


Allied health professions are those professions that are not medical, nursing, or pharmacy. Examples include 
Emergency Medical Technician, surgical technician, and medical coding specialist. 


https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-supplement- 
evaluation-design-impact-analysis-plan 
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Executive Summary 


Low-income workers with only a high school education face poor and declining employment 
prospects (Pew Research Center 2014). Postsecondary training, often at community colleges, 
offers one strategy for improving this population’s education and employment opportunities, 
especially if targeted to occupations where there is high and growing demand for skilled 
workers (Capelli 2014; Conway and Giloth 2014; Holzer 2015). How to facilitate a better match 
between the nation’s need for a skilled workforce and the needs of low-income adults for 
employment is a topic of great interest to policymakers, workforce development organizations, 
educators, and other key stakeholders. 


VIDA Program 


This report provides evidence on the implementation and early impacts of one promising effort 
to meet the needs of low-income students and local employers for skilled workers, the Valley 
Initiative for Development and Advancement (VIDA). Founded in 1995, VIDA, a community- 
based organization created through a partnership of faith-based leaders and the business 
community in the Lower Rio Grande Valley of Texas has a mission “to formulate new 
institutional relationships in the Rio Grande Valley that simultaneously address employers’ 
needs for skilled workers and link the area’s unemployed and underemployed with high skilled, 
high wage jobs identified in the region.”° VIDA’s primary goal is for participants to graduate 
with an associate’s degree or industry-recognized certificate in a high demand occupation and 
achieve living-wage employment in their area of study. 


The report includes findings from the implementation study and the impact study. The former 
showed that VIDA continued to implement its program as designed. Participants received 
assistance with tuition and school-related expenses as well as intensive counseling and personal 
guidance. 


The impact study showed that over an initial 24-month follow-up period, compared to a 
randomly assigned control group that could not access the program, VIDA treatment group 
members: 


e Earned more college credits, 
e Enrolled in college full-time at a greater rate, and 


e Earned more college credentials. 


VIDA supports full-time enrollment of individuals in occupational training programs leading to 
certificates and degrees in occupations with high local employer demand. Participants must be 


3 See VIDA website at http://www. vidacareers.org/whowe.htm. 
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unemployed, underemployed, meet federal poverty income levels, or be on public assistance. 
Additionally, they must be residents of the Rio Grande Valley, 18 years or older, and eligible to 
work in the United States. 


To support participants’ successful completion of training, VIDA provides comprehensive and 
intensive counseling services and substantial financial assistance at a cost of approximately 
$13,750 per participant over an average of two and a half years. VIDA’s primary program 
features are: 


e Required full-time enrollment in certificate programs, associate’s degree programs, or 
the final two years of bachelor’s degree programs. VIDA staff conduct an initial 
assessment of each applicant to determine his or her ability to commit to the program. 
The assessment accounts for finances, personal circumstances, and the need for VIDA 
services. 


e Weekly mandatory group or individual case management and counseling sessions 
conducted at students’ colleges by experienced VIDA Counselors with degrees in 
education, social services, psychology or a related field provide information on how to 
succeed in college and employment, identify problems early, and provide social support. 


e Wrap around support services for tuition, books, and other needs, such as 
transportation assistance calculated on school attendance to reduce the cost of 
obtaining a certificate or degree. 


e The “College Prep Academy,” a 16-week, accelerated, full-time basic skills (“bridge”) 
program for those who are not college ready but who have tenth grade skill levels or 
better. The course meets daily to prepare individuals to pass college entrance exams. 


e Regular assessment of local labor markets to identify occupations where support for 
training from VIDA is most likely to promote employment. 


Together these features and other less central program components provide a substantial set 
of services for which a participant must meet a demanding set of requirements. Their aim is to 
move students quickly through standard college coursework in order to obtain a certificate ora 
degree and secure employment. 


Pathways for Advancing Careers and Education (PACE) Evaluation 


Abt Associates and its partners are evaluating VIDA as part of the Pathways for Advancing 
Careers and Education (PACE) evaluation. Funded by the Administration for Children and 
Families (ACF) in the Department of Health and Human Services, PACE is an evaluation of nine 
programs that include key features of a “career pathways framework.” 


The career pathways framework guides the development and operation of programs aiming to 
improve the occupational skills of low-income adults, primarily older nontraditional students, 
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by increasing their entry into, persistence in, and completion of postsecondary training. Central 
to accomplishing these outcomes, the framework describes strategies for overcoming barriers 
to education and training that these students can face. Key features of programs within this 
framework include: a series of well-defined training steps, promising instructional approaches 
targeted to adult learners, services to address academic and non-academic barriers to program 
enrollment and completion, and connections to employment. 


The VIDA evaluation includes an implementation study that examines the design and operation 
of the program as well as participation patterns of students enrolled, and an impact study that 
uses an experimental design to measure differences in educational and employment outcomes 
for a sample of 478 individuals randomly assigned to a treatment group that could participate 
in VIDA or a control group of 480 that could not.* Using data from baseline surveys, a follow-up 
survey, administrative records, in-depth participant interviews, and site visits, this report 
provides the results from the implementation study and describes the early impacts of the 
program (24 months after random assignment) on education, training and employment.” 


Key Findings 


This summary documents findings from the implementation study and early findings from the 
impact study at 24 months after randomization, including receipt of college credits received 
since random assignment, the primary outcome used to assess the early effects of VIDA. 


Implementation Study 


The implementation study examined how VIDA’s counseling and financial support services were 
implemented, the degree to which treatment group members enrolled in and completed 
occupational training programs, and the receipt of counseling and other services by the 
treatment group relative to the control group. 


e Participants reported benefiting significantly from the counseling provided to them by 
VIDA, and qualitative data suggests that treatment group members received a substantial 
dose of these services. 


In interviews, participants discussed the importance of the VIDA Counselors to their experience 
in the program. Participants said that they believed the Counselors understood what they were 
going through, provided them with encouragement when school felt difficult or personal 

challenges arose, and offered strategies and guidance. Qualitative information from PACE team 


Random assignment ensures that the treatment and control groups will be equivalent in their observed and 
unobserved characteristics, and that any systematic differences in their subsequent outcomes can be 
attributed to the treatment group having access to program services. 


> The analysis plan was posted on the ACF website at http://www.acf.hhs.gov/opre/resource/pathways-for- 


advancing-careers-and-education-supplement-evaluation-design-impact-analysis-plan. Outcomes were also 
registered on the Open Science Framework site. https://osf.io/bpm8u/. 
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program observations, interviews with staff, and interviews with participants, suggests the 
majority of treatment group members received a substantial dose of individual and group 
counseling. 


e Almost all participants (99 percent) received VIDA’s assistance with tuition and expenses 
related to attending and completing school, and VIDA provided an average of almost 
$7,000 in direct financial assistance per participant within 24 months of random 
assignment. 


VIDA carefully calculates the amount of financial support it will provide each participant, taking 
into consideration his/her household finances, the type of program into which the participant is 
enrolling, and the length of time it expects the student to participate in the program. In the 

24 months after random assignment, VIDA spent funds on 98 percent of treatment group 
members.° In the two years after random assignment, VIDA spent an average of $6,808 per 
participant (for the 98 percent of treatment group members for whom they expended any 
financial resources). Of the participants who received financial assistance, almost all 

(99 percent) received assistance with tuition and related expenses of attending and completing 
training, which averaged $4,861 per participant. Eighty-nine (89) percent of participants had 
assistance paying for books, at an average of $910 per participant, and 88 percent received at 
an average of $835 per participant for transportation assistance for going to and from school. 
Thirty (30) percent of participants had help paying for uniforms or interview clothing, because 
VIDA views these as necessary expenses for participation in training and attaining a job. Less 
common among the types of financial support related to training was assistance for tools 

(23 percent), school supplies (19 percent), and tutoring (3 percent). 


e VIDA staff reported that the program was a set of obligations and incentives and not just 
a set of services; participant interviews reflected this perspective, too. 


VIDA commits to providing a high degree of financial and personal support, but the participant 
must agree to meet VIDA’s program requirements, including maintaining full-time enrollment in 
school, achieving passing grades, and participating in weekly counseling sessions. VIDA staff 
articulated the view that financial support was more than a service; it was a key incentive for 
both attracting applicants to the program and fostering retention in the program and in 
training. Meanwhile, VIDA Counselors provide participants with the tools to address academic 
and personal challenges that otherwise might derail their participation in training. 


In this report, “participants” constitute the 98 percent of treatment group members for whom VIDA provided 
at least one direct financial assistance payment. 
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e Nearly all treatment group members (98 percent) participated in occupational training 
and the vast majority (91 percent) completed college credits. About two-thirds 
accumulated 30 or more credits within the follow-up period. 


Almost all treatment group members (98 percent) participated in any education or training 
activity, which includes both basic skills education in the College Prep Academy and 
occupational training. Ninety-seven percent of all treatment group members participated in 
occupational training, including those who advanced from the College Prep Academy into 
college-level programs of study. Ninety-one percent of treatment group members earned at 
least one college credit, and 66 percent completed 30 or more college credits (by some 
standards the equivalent of a year of full-time college). 


e The College Prep Academy prepared participants for entry into college-level occupational 
training programs, although VIDA was not able to increase enrollment in the program or 
expand to additional colleges as intended. 


Upon entering the program, nine percent of treatment group members needed instruction in 
math, reading, and writing in order to pass the college entrance exam. These students generally 
had a high school diploma or GED as their highest level of education at baseline, and started in 
the College Prep Academy. The majority of participants (85 percent) enrolled in occupational 
training after completing the College Prep Academy. Though the College Prep Academy had 
historically enrolled a small proportion of participants, VIDA planned to increase enrollment in 
the Academy during PACE. Its analysis of educational attainment levels in the region, where 
approximately one-fifth to one-quarter of residents held a high school diploma or equivalent 
but had not attended college, suggested that there was a demand for accelerated remedial 
education. However, the program was unable to expand the College Prep Academy as planned. 
VIDA found it more difficult to recruit interested and eligible participants than expected, and 
they encountered challenges expanding the Academy to a second college and discontinued this 
new partnership. 


e Persistence in occupational training programs was substantial, with slightly more than 
four in 10 treatment group members still attending training at the end of the 24-month 
follow-up period. 


Two years after random assignment and enrollment into VIDA’s program, 42 percent of the 
treatment group was still attending training. Among those still attending, about half were still 
working towards a credential while the other half had obtained a credential, suggesting they 
continued in their educational pathway to attain further certificates or degrees. 
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e Training in nursing and allied health professions were the programs most commonly 
attended by treatment group members, followed by programs in education, social 
services, and specialized trades. 


Of treatment group members who engaged in occupational training, three-quarters 
participated in nursing or allied health training, ’ according to VIDA administrative records. Half 
of those enrolled in health training pursued nursing programs. One-quarter enrolled in Licensed 
Vocational Nurse (LVN) certificate programs, 19 percent enrolled in Associate’s Degree in 
Nursing (ADN) programs, and eight percent in a pathway program that advanced participants 
from LVN to ADN. Other health programs of training, attended by one-quarter of treatment 
group members, included certificate and associate’s degree programs for Radiology Technician, 
Patient Care Assistant, Respiratory Therapist, Surgical Technician, Health Information 
Technology, and a Bachelor’s of Science in Nursing (BSN). After health training, VIDA 
participants most commonly enrolled in programs in education and social services (14 percent), 
including Social Worker, Criminal Justice Corrections Officer, Teacher, and Sign Language 
Interpreter. Eleven percent of treatment group members participated in specialized trades 
programs such as Heating, Ventilation, Air Conditioning, and Refrigeration Technician and 
Automotive Technician. Nine percent of participants pursued business-related programs 
including Business Management Technology, Office Management, and Accounting, and seven 
percent enrolled in technology programs such as Biomedical Equipment Technology, Computer 
and Telecommunications Technology, and Aviation Maintenance Technology. 


Impact Study 


The impact study reports estimates of VIDA’s early impacts on educational attainment, career 
progress, and non-economic outcomes. The main estimates cover impacts over a 24-month 
period after random assignment for the full sample of 958 randomly assigned participants. The 
research team also explored impacts for a longer follow-up period—36 months—for the 
roughly three-quarters of the sample who enrolled through May 31, 2013 (who could be 
observed for a longer time). 


Prior to conducting these analyses, the research team categorized hypotheses into three 
groups: confirmatory, secondary, and exploratory. The confirmatory hypothesis focuses on the 
outcome which best represents whether the program is on track to have positive effects on its 
main goals consistent with the expected direction of the effect given the timing of the follow 
up. Secondary hypotheses include a limited set of additional outcomes which, like the 
confirmatory outcome, are consistent with expected effects within the period covered by the 
study report. Exploratory hypotheses include additional outcomes whose direction and timing 
are less certain. 


7 Allied health professions are those professions that are not medical, nursing or pharmacy. Examples include 


Emergency Medical Technician, surgical technician, and medical coding specialist. 
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e VIDA significantly increased the total number of college credits earned (the confirmatory 
outcome). 


As Exhibit ES-1 shows, over a 24-month period, treatment group members earned 33.1 credits 
compared with 27.5 credits for the control group, an increase of 5.6 credits, or roughly two 
typical courses completed successfully. The 27.5 credits for control group members reflects the 
fact that they had access to the same college courses as treatment group members and that 
applicants were often already enrolled when they applied for VIDA’s services. These impacts 
are among the largest reported to date from random assignment tests of programs aiming to 
increase college success among low-income individuals.® In addition, an analysis of the pattern 
of this effect over time showed growing impacts on credits earned over each six-month period 
after random assignment. 


e The program increased rates of full-time college enrollment and enrollment more 
generally. 


VIDA increased full-time enrollment in college, an outcome that is central to its theory of 
change. As Exhibit ES-1 shows, VIDA increased full-time enrollment in each six-month 
increment during a 24-month period by between about five and 11 percentage points, and by 
11 points over the full period. 


* The research team’s review of the literature, as well as consulting with others, indicates that only the MDRC 


evaluation of the CUNY ASAP program has demonstrated (slightly) larger impacts on credits and credentials 
(Scrivener et al. 2015). See Section 6.1 of the full report for detail. 
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Exhibit ES-1. Early Impacts on Education Outcomes (Confirmatory and Secondary Hypotheses) 


Treatment Control Impact Standard 
Outcome Group Group (Difference) Error p-Value 
Confirmatory Outcome 
Total college credits earned within 24 months of 33.1 27.5 +5.6*** 14. <.001 
randomization (average) 
Total college credits earned by end of successive periods after randomization 
By month 6 11.2 9.6 +1.6"* 0.5 <.001 
By month 12 21.3 18.0 +3," 0.7 <.001 
By month 18 28.2 23.4 +4,Q%** 1.0 <.001 
By month 24 33.1 27.5 +5.6*** 11 <.001 
Enrolled full time in academic/technical college courses during months (%) 
Months 1-6 64.6 55.0 +9.6"* 3.0 <.001 
Months 7-12 51.9 40.8 +11,0"* 3.1 <.001 
Months 13-18 31.3 24.4 +6,9*** 2.9 .008 
Months 19-24 20.8 16.2 +4,.6* 2.6 .036 
In any month 1-24 75.7 64.8 +10.9** 2.8 <.001 
Total months of college enrollment (average): 
Full time 6.89 5.61 +128" 0.36 <.001 
Any 15.38 13.14 +2,24%* 0.46 <.001 
Receipt of a college credential (%): 
Level 1 certificate 12.1 10.6 +15 2.0 226 
Level 2 certificate 15.9 12.7 +3.2* 2.1 .067 
Any certificate 28.0 23.1 4.9** 2.8 .040 
Associate's degree 26.1 22.9 +3.2 2.7 116 
Bachelor's degree 2.7 2.1 +0.6 0.9 267 
Any degree 28.8 25.0 +3.8* 27 .080 
Any credential 53.3 45.0 +8,3*** 3.0 .003 
Enrolled in any summer school within 24 months 73.2 56.5 +16.7*** 2.9 <.001 
of random assignment (%) 
Total hours of occupational training at a college 11.8 14.6 -2.8 8.1 635 
that does not grant degrees 
Received a credential from: (%) 
A college within 24 months 53.3 45.0 +8.3"** 3.0 .003 
Another education or training institution by 0.6 0.9 -0.2 0.6 646 
time of survey 
A licensing/certification body by time of survey 34.4 35.0 -0.6 3.4 567 
Any of the above 62.2 55.5 +6.7* 3.4 024 
Sample size 4 478 480 


SOURCE: Abt Associates calculations based on data from VIDA partner college records and the PACE short-term follow-up survey. 
NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between treatment and control groups, are 
summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

* The numbers in this row are the sample sizes for estimates based on college records for the full sample. These are all estimates 
except for the last two panels. In the last two panels (total hours and credentials by place), estimates for activity at a college also 
reflect college records for the full sample, but all other estimates in these two panels (including those for activity at any source) 
reflect the subsample who responded to the PACE follow-up survey, 434 treatment and 373 control group members. 
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These effects were significant at the five percent level, with most at the one percent level or 
smaller. About three-quarters of treatment group members were enrolled full time at some 
point. Viewed in terms of months of enrollment, VIDA increased full-time enrollment by about 
1.3 months and any enrollment by about 2.2 months (both significant at the one percent level) 
within the 24-month follow-up period. 


e VIDA significantly increased college credentials earned. 


VIDA significantly increased the percentage of individuals who earned a college credential by 
eight percentage points: 53 percent of treatment group members earned a credential 
compared to 45 percent of control group members (significant at the one percent level). Effects 
were largest for those who received a certificate, about five percentage points, with about two- 
thirds of this effect due to a significant impact on certificates that require 43 to 59 credits 
(known as Level 2 certificates). There was also a significant, positive effect (significant at the ten 
percent level) on those receiving academic degrees, about four percentage points, and this 
effect was almost entirely due to receipt of associate’s degrees. VIDA did not affect receipt of 
non-college degrees or non-academic licensing/certificates. 


e Educational impacts seen for a subgroup of early enrollees (followed for 36 months) 
suggest that VIDA’s full sample impacts will grow over the next year. 


Availability of an additional year of college data for sample members who had enrolled in VIDA 
by the end of May 2013—somewhat more than three-quarters of the full sample—enables 
analyses of a 36-month follow-up period. These analyses provide a suggestive look into 
potential future impacts. VIDA had positive effects on college credits earned, full-time 
enrollment in college (persistence), and receipt of a college credential (college success) at 

36 months. Further, the increases all were larger in magnitude than those at 24 months. The 
increase in college credits earned was seven credits over 36 months for the early cohort 

(39.4 credits versus 32.5 credits).The research team anticipates that the 36-month effects are 
likely to hold for the full sample for two reasons. First, the early cohort represents over three- 
quarters of the full sample, and, thus, in the full sample analysis will carry over three times the 
weight of the remaining roughly one quarter of the treatment group. Second, the outcome 
levels and point estimates for the full sample and the early cohort are extremely close in 
magnitude over the first 24 months, a period over which both are observed. 


Next Steps in the VIDA Evaluation 


This report on VIDA focuses on the implementation of the program and its early effects on 
participants’ education and training. The findings on VIDA’s impacts on college outcomes over 
two years of follow-up represent some of the largest effects to be established in random 
assignment evaluations of promising postsecondary programs. The key program goal examined 
at 24 months after random assignment to the program or a control group was receipt of college 
credits received since study enrollment. This emphasis on education outcomes reflects VIDA’s 
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theory of change and expectations that many participants in the program would still be 
engaged in training at the end of the 24 months. 


The next VIDA report will cover a 36-month to 42-month follow-up period. It will take a 
more systematic look at program effects on participants’ economic outcomes for a period 
when these are expected to occur. The report will examine employment outcomes, such 
as average rate of employment and average earnings over successive follow-up quarters, 
and job characteristics, such as occupation, hourly wage, receipt of benefits, and career 
progress. Thus, it will begin to answer whether the occupational training gains that VIDA 
achieved after 24 months will translate into economic gains in the workplace in the longer 
term. An analysis at approximately 72 months after random assignment will estimate 
long-term effects of the program. Key research questions for these future analyses 
include: 


e Will VIDA’s educational effects remain stable, grow, or shrink? 


e Will VIDA’s positive educational effects translate into positive impacts on employment 
outcomes in the longer term, consistent with the program’s theory of change? 


e Does VIDA have other positive effects on treatment group members and their families? 


e Is VIDA cost beneficial? 
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1. Taldmelelereaa(ela 


Low-income workers with only a high school education face poor and declining employment 
prospects (Pew Research Center 2014). Postsecondary training, often at community colleges, 
offers one strategy for improving this population’s education and employment opportunities, 
especially if targeted to occupations where there is high and growing demand for skilled 
workers (Capelli 2014; Conway and Giloth 2014; Holzer 2015). How to facilitate a better match 
between the nation’s need for a skilled workforce and the needs of low-income adults for 
employment is a topic of great interest to policymakers, workforce development organizations, 
educators, and other key stakeholders. 


Research indicates that making such a match is not easy, as many low-income, low-skilled 
adults face considerable barriers to completion of postsecondary education. Many such adults 
are “nontraditional” students—that is, often they are older, are parents, lack adequate basic 
academic skills, and have few economic resources (National Center for Education Statistics). 
Research further shows that on average, nontraditional students fare worse in postsecondary 
settings than do traditional students (Visher et al. 2008; Cooper 2010; Goldrick-Rab and 
Sorensen 2010). Institutions often assign students who need to improve their basic academic 
skills to developmental (remedial) education; many of these students never progress beyond it 
(Rutschow and Schneider 2011). Others drop out due to financial setbacks or difficulties 
juggling school, work, and family responsibilities. Some have difficulties navigating the college 
environment, including course sequences and financial aid. Many have difficulty meeting 
academic standards (Bridges to Opportunity Initiative 2008). 


This report provides early evidence on the implementation and impacts of one promising effort 
to meet the needs of low-income students and local employers for skilled workers, the Valley 
Initiative for Development and Advancement (VIDA). Established in 1995, VIDA is a non-profit, 
community-based organization, created through a partnership of faith-based leaders and the 
business community of the Lower Rio Grande Valley, Texas. VIDA has a mission “to formulate 
new institutional relationships in the Rio Grande Valley that simultaneously address employers’ 
needs for skilled workers and link the area’s unemployed and underemployed with high skilled, 
high wage jobs identified in the region.”° VIDA’s developers adapted a model originating with 
the Project QUEST program, initiated in San Antonio in the early 1990s. As further evidence of 
the model’s potential replicability, eight other community-based organizations also have 
adapted it.”° 


9 VIDA website, http://www.vidacareers.org/whowe.htm. 


*0 For more about Project QUEST, see http://www.questsa.org. 
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To accomplish its mission, VIDA supports full-time enrollment of adults in occupational 
programs at local colleges with which it has a partnership. The programs lead to certificates and 
degrees in occupational areas with high local employer demand for graduates. Participants 
must be unemployed, underemployed, meet federal poverty income levels, or be on public 
assistance. Additionally, they must be residents of the Rio Grande Valley, 18 years or older, and 
eligible to work in the United States. The great majority of participants have college-level 
academic skills before enrolling in VIDA. To support successful completion of training, VIDA 
provides its participants with comprehensive and intensive counseling services and substantial 
financial assistance, at a cost of approximately $13,750 in direct financial supports and services 
per participant (over an average of two and a half years). 


VIDA’s primary program features are: 


e Required full-time enrollment in certificate programs, associate’s degree programs, or 
the final two years of bachelor’s degree programs. An applicant’s ability to make this 
commitment is determined by an initial assessment conducted by VIDA staff that 
accounts for finances, personal circumstances, and the need for VIDA services. 


e Weekly mandatory group or individual case management and counseling sessions to 
provide information on how to succeed in college and employment, identify problems 
early, and provide social support. The sessions are conducted at students’ colleges by 
experienced VIDA Counselors with degrees in education, social services, psychology or a 
related field. 


e Provision of wrap around support services for tuition, books, and other needs, such as 
transportation assistance calculated on school attendance to reduce the cost of 
obtaining a certificate or degree. 


e The “College Prep Academy,” a 16-week, accelerated, full-time basic skills (“bridge”) 
program for those who are not college ready, but who have tenth grade skill levels or 
better. The course meets daily to prepare individuals to pass college entrance exams. 


e Regular assessment of local labor markets to identify occupations where support for 
training from VIDA is most likely to promote employment. 


Along with other less central program components, these features together provide a 
substantial set of services for which a participant must meet a demanding set of requirements. 
VIDA’s aim is to move students quickly through standard college coursework. 


Abt Associates and its partners are evaluating VIDA as part of the Pathways for Advancing 
Careers and Education (PACE) evaluation. The evaluation of the VIDA program includes both an 
implementation study to examine its design and operation and an impact study that relies ona 
random assignment research design to estimate the impacts of access to VIDA on its students’ 
education and training, employment, and other outcomes. 
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This report describes VIDA implementation and early impact findings on participant outcomes 
within a follow-up period of approximately 20-24 months.** This chapter describes the PACE 
evaluation, summarizes findings from the research literature regarding the principle ways in 
which VIDA augments existing college training, and provides a roadmap to the rest of the 
report. 


1.1. Pathways for Advancing Careers and Education (PACE) Evaluation 


Funded by the Administration for Children and Families (ACF) within the U.S. Department of 
Health and Human Services, the PACE evaluation is a 10-year study of nine programs that 
include key features of a “career pathways 

framework.” Initiated in 2007, PACE Programs in PACE 

represents the first large-scale, multi-site e Bridge to Employment in the Health Care Industry at San 
experimental evaluation of career pathways Diego Workforce Partnership, San Diego, CA 
programs. 


Carreras en Salud at Instituto del Progreso Latino, 
Chicago, IL 
Many of the key ideas in the career 


Health Careers for All at Workforce Development Council 


pathways framework guide the development of Seattle-King County, Seattle, WA 

and operation of programs that aim to e Pathways to Healthcare at Pima Community College, 
improve the occupational skills of low- Tucson, AZ 

income individuals, primarily older e Patient Care Pathways Program at Madison College, 


nontraditional students, by increasing their Madison, WI 
entry into, persistence in, and completion of 
postsecondary training. Central to 
accomplishing these improved outcomes, 


Valley Initiative for Development and Advancement 
(VIDA), Lower Rio Grande Valley, TX 


Washington Integrated Basic Education and Skills 
Training (I-BEST) program at three colleges (Bellingham 


the framework describes signature Technical College, Whatcom Community College and 
strategies for overcoming the barriers that Everett Community College), Washington State 
nontraditional, occupational students often e Workforce Training Academy Connect at Des Moines 


face. For example, key features of many Area Community College, Des Moines, 1A 
programs within this framework include a Year Up (Atlanta, Bay Area, Boston, Chicago, National 
series of well-defined training steps, Capital Region, New York City, Providence, Seattle) 
promising instructional approaches, 

supportive services, and connections to employment (Fein 2012). 


The career pathways framework suggests that programs may be more effective to the degree 
that they include multiple components, reflecting the observation that nontraditional students 


“The average completion of the initial follow-up survey for VIDA occurred about 20 months after random 


assignment, and college records data are available for the full VIDA sample for 24 months. 


More information on the PACE evaluation can be found on ACF’s Office of Planning, Research and Evaluation 


website: http://www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers-and-education. 
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face multiple barriers to success and that addressing only a single one is unlikely to substantially 
improve their educational and employment prospects. In the case of VIDA, the program 
accommodates both participants with pre-college levels of education and those who are college 
ready; includes a thorough assessment to determine both financial and support service needs 
and the ability of participants to attend college full time; includes intensive counseling services 
in group and individual formats; and provides extensive financial assistance tailored to 
individual need. 


The career pathways framework is flexible, however, and not a specific program model. For 
example, though some PACE programs created special instructional approaches, others like 
VIDA rely primarily on the standard instructional offerings at local colleges.’ Thus, program 
components and their implementation can vary greatly. 


Reflecting this diversity, each of the nine programs in the PACE evaluation represents a 
different program model. All share some program components that are part of the career 
pathways framework, but each also has distinct and unique elements, reflecting the target 
populations, occupational training offered, and industries of focus. Because of this variation, 
PACE evaluates and reports findings for each program individually. 


The central goal of the PACE evaluation is to determine the effectiveness (impact) of each of 
the nine programs using a common evaluation design and conceptual framework. The most 
critical element of the evaluation design is random assignment of eligible applicants either to a 
treatment group that can access the career pathways services or to a control group that 
cannot. Random assignment ensures that the study’s treatment and control groups will be 
equivalent in their observed and unobserved characteristics, and that any systematic 
differences in their subsequent outcomes can be attributed to the treatment group having 
access to program services. Consistent with the framework and the career pathways theory of 
change guiding the PACE evaluation (described in Chapter 2), the key outcomes for which the 
PACE study estimates effects are in the domains of education and training and employment, 
although the study also estimates effects in other areas, such as family well-being. 


The PACE implementation and early impact program reports analyze outcomes over 
approximately 18 to 24 months since random assignment. The impact analyses rely primarily 
on surveys and college records for individuals in the treatment and control groups. Future 
reports will analyze outcomes approximately three years and six years after random 
assignment.“ These latter reports will also include benefit-cost studies. 


8% The exception for VIDA is the College Prep Academy that VIDA designed but for which its college partners 


provide instruction. 


“These reports will be part of the Career Pathways Intermediate Outcomes and Career Pathways Long-term 


Outcomes Projects, respectively. 
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ACF and the PACE research team selected VIDA because the program implements promising 
features of the career pathways framework and it was of sufficient scale to generate a research 
sample large enough to support a standalone impact study.” In addition, VIDA has a stable 
program that had been implemented for more than a decade at the time of its selection into 
PACE and is a replicable model that operates successfully in multiple locations. 


1.22 ViIDAinthe Context of Related Research 


As described earlier, VIDA complements local college occupational training with five primary 
program features. This section briefly describes what research suggests about the effectiveness 
of these features in improving outcomes. 


Prior to PACE, there have been four evaluations of the Project QUEST model: three of Project 
QUEST itself and one of the Capital Idea program in Austin, Texas, which is based on the Project 
QUEST model. The strongest evidence in the four studies comes from a random assignment 
study of Project QUEST which showed very large positive effects on health care certificates of 
26 percentage points and annual earnings of roughly $5,000 six years after random assignment 
(Elliott and Roder 2017).*° Although the Project QUEST results are generally encouraging given 
that Project QUEST and VIDA share a common model, there are a number of reasons why these 
findings do not provide the best context for understanding the current PACE results for VIDA: 


e The primary observation period for this report is 20-24 months after random 
assignment, whereas the Project QUEST evaluation summarizes effects over six years, 
making comparisons inappropriate; 


e The Project QUEST study enrolled only individuals who were not currently enrolled in 
college, whereas VIDA enrolled many individuals who were already attending one of 
their partner colleges; 


e The Project QUEST evaluation enrolled individuals who were seeking a credential for a 
health care profession, but PACE study participants could enroll in any college program 
that VIDA supported; and 


e Project QUEST primarily targeted certificate programs, whereas the most common 
academic goal for VIDA participants was an associates’ degree. 


Because of these important differences, a better research context for understanding the 
current PACE results for VIDA are recent findings concerning a program that includes three 


* The criterion for “promising” included positive empirical evidence of effectiveness for key components of the 


program or systematic, well-developed approaches to overcoming identified barriers to student success. 


*© The other three studies have very weak non-random assignment designs (pre/post or a poorly matched 


comparison group) and very likely overstate the size of any positive effects (Osterman and Lautsch 1996, 
Rademacher, Bear and Conway 2001 and Smith, Christensen and Cumpton 2015). 
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central elements of VIDA—(1) a requirement for full-time enrollment, (2) mandatory and 
intensive advising, and (3) substantial financial assistance —in a random assignment study 
demonstrated very large effects on rates of completion of two-year degrees. 


The City University of New York Accelerated Study in Associate Programs (ASAP) approximately 
doubled graduation rates over a three-year, follow-up period (Scrivener et al. 2015). In 
addition, a replication of the program in Ohio, although at an earlier stage, also suggests 
potentially large effects on educational progress (Sommo and Ratledge 2016). Notably, the 
ASAP program targets younger general education students with some basic skills remediation 
needs, whereas VIDA’s primary population is older occupational students, only a small 
percentage of whom have basic skills deficits. Thus, whether this combination of features is 
effective for VIDA’s population is an important as yet unanswered question. 


There is also evidence supporting some of VIDA’s individual program features. Studies have 
shown that typically community college advisors have very large student-to-advisor ratios, 
typically more than 1,000 to 1 (Gallagher 2011). By contrast, VIDA Counselors had a caseload of 
about 50-55 students. Several rigorous studies have demonstrated that expanding the typical 
services with more intensive advising can lead to greater persistence in education, although 
sometimes only for the short term (Bettinger and Baker 2011; Scrivener and Weiss 2009). In 
addition, a large body of evidence indicates that insufficient personal resources are a barrier to 
entry and completion of education and training for low-income students and that financial 
assistance can increase postsecondary attendance and persistence (Deming and Dynarski 2010; 
Dynarski and Scott-Clayton 2013). 


VIDA also encouraged participants whose basic skills were too low to enter occupational 
training directly to enroll in its bridge program, the College Prep Academy. In the absence of 
the College Prep Academy, such students would need to enroll in and complete appropriate 
courses in a developmental education sequence in order to enroll in college-level courses. 
Evidence indicates that most students referred to developmental education never enroll in 
college-level courses (Bailey, Jeong, and Cho 2010). However, evidence on the effectiveness of 
bridge programs is mixed. Some research suggests that compressing developmental education 
into shorter periods, as the College Prep Academy does, can improve outcomes for low-skilled 
students (Zeidenberg, Cho, and Jenkins 2010, Rutschow and Schneider 2011), but other studies 
have found no or only short-term effects (Fein and Beecroft 2006; Barnett et al. 2012). 


1.3. Structure of This Report 
The organization of the remainder of this report is as follows: 


e Chapter 2 presents the VIDA evaluation’s conceptual framework and research 
questions; details the evaluation design; describes the study sample; and summarizes 
the evaluation’s data sources. 
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e Chapter 3 describes VIDA’s context, its administrative structure, and its design features. 


e Chapter 4 describes the implementation study findings, including case management and 
advising, participation in training, financial support, and employment supports provided 
by the program. 


e Chapter 5 presents the impact study findings, focusing on the effect of access to VIDA 
on receipt of college credits over a 24-month period—as well as a series of other 
educational and life outcomes. 


e Chapter 6 discusses what about the program may have led to the impact findings and 


what to expect in the longer term. 


The appendices provide additional details about the baseline data (Appendix A); college records 
data, a key source for the impact study (Appendix B); survey-based outcomes (Appendix C); and 
the approach to outliers (Appendix D). 
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2. PACE Evaluation Design and Data Sources 


This chapter describes the larger PACE evaluation design and its application to VIDA. It begins 
with a discussion of the PACE career pathways theory of change and the research questions 
that the theory of change implies. It then briefly describes the evaluation design and analysis 
procedures for the impact study, including the random assignment process and the outcome of 
that process. A brief description of the implementation study analysis follows.’” Finally, the 
chapter summarizes the main data sources for the implementation and impact studies. 


2.1 Career Pathways Theory of Change 


The career pathways theory of change guides both the implementation study (i.e., it identifies 
which aspects of program services are expected to affect outcomes) and the impact study 

(i.e., it identifies which outcomes the program is expected to affect). The theory of change also 
generates key hypotheses about the direction of expected effects that the impact evaluation 
will test for statistically significant differences between the treatment and control group. © In 
addition, the theory of change implicitly assumes time horizons by which the program is 
expected to have effects, and thus the theory determines the key outcomes at any particular 
time of follow-up. 


Exhibit 2-1 (below) depicts the PACE career pathways theory of change, as applied to VIDA.*? It 
shows how a program (inputs) is hypothesized to produce effects on intermediate outcomes, 
which in turn will lead to effects on main outcomes. Effects on intermediate outcomes are 
expected earlier than effects on main outcomes, but the exact timing depends on particular 
features of the program, such as the length of occupational training and what, if any, steps 
precede it. In addition, because effects on intermediate outcomes may persist over time, and 
even possibly be amplified over time by impacts on main outcomes, the study will also measure 
them at later points in time. 


“The research team developed a detailed evaluation design report for the PACE evaluation, including the 


evaluation of VIDA: http://www.acf.hhs.gov/programs/opre/resource/pathways-for-advancing-careers-and- 
education-evaluation-design-report. 


The implementation study describes the set of services that students in the treatment group experienced and 


how those services differed for the control group. The impact study focuses on estimates of the effects of the 


program on intermediate and main outcomes. 


*° See Fein (2012) for an extended description of the framework. 
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PROGRAM INPUTS 


ORGANIZATION 


+ VIDA 


+ VIDA counselors and 
management staff 

+ Economic development 
corporations provide 
funding, information on 
demand occupations 

+ Funding from 
foundations and local 
governments 


* Colleges and 
universities 


PARTICIPANTS 


+ At least 18 years of age 

+ Unemployed, 
underemployed, or on 
public assistance 

+ Aresident of Cameron, 
Hidalgo, Starr, or 
Willacy County 

+ High school diploma or 
GED 

+ Interest in high-demand 
occupations identified 
by VIDA 


PROGRAM COMPONENTS 


ASSESSMENT 


* College entrance exam 
scores or math, reading 
and writing assessment 
completed during 
Placement Session 

* Career Decision-Making 
System assessment 

* Individual in-person 
interviews by VIDA 
counselors and 
senior staff (both at 
application and after 
admission to VIDA) 

+ Financial needs 
assessment 


INSTRUCTION 


*16-week College Prep 
Academy to prepare 
participants at grade 
levels 10-12 for college 
entrance exams 


+ Enrollment in local 
colleges’ certificate 
and associate’s degree 
programs related to 
in-demand occupations 
(less commonly, 
enrollment in last two 
years of bachelor’s 
degree programs) 


Abt Associates 


SUPPORTS 


* Dedicated counselor 
to provide weekly 
counseling on academic 
and personal issues in 
group and individual 
settings 

+ Financial assistance 
with tuition, books, 
course-related 
materials, childcare 
and transportation, and 
emergency assistance 

« Referrals to needed 
supports 

+ Required community 
service 


EMPLOYMENT 


+ Group counseling 
sessions on job search 
preparation and skills 

* Support of credentials 
in high-demand 
occupations only 

+ Follow-up post- 
completion to identify 
employment status; 
help unemployed with 
resume 


Exhibit 2-1. Career Pathways Theory of Change (for VIDA) 


INTERMEDIATE OUTCOMES 


GENERAL (21ST CENTURY) COMPETENCIES 


+ Improved basic academic skills for College Prep 


Academy participants 
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MAIN OUTCOMES 


POSTSECONDARY ATTAINMENT 


+ Enrollment in certificate or degree program 
tied to in-demand occupation 


+ Improved psycho-social skills (persistence, * Credits 


academic self-confidence, sense of belonging) 


SPECIFIC COMPETENCIES 


+ Improved occupational skills in desired area 


CAREER KNOWLEDGE 


+ Increased awareness of steps needed to reach 


career goals 


+ Increased knowledge of labor market 


RESOURCES 


* Constraints addressed through financial 
assistance and counseling services 


LIFE CHALLENGES 


+ Reduced financial hardship 


* Credentials: completion of certificate or 
associate’s degree program (less commonly, 
bachelor’s degree program) 


SUCCESSFUL IN CAREER-TRACK 

EMPLOYMENT 

+ Attainment of industry licenses, if applicable 

* Employment in an in-demand occupation with 
benefits and living wage 

+ Employment progression and possibility for 
further education 


OTHER LIFE OUTCOMES 


* Improved individual and family well-being 
+ Improved family economic status 


+ Ability to address and mitigate stressors 


LOCAL POSTSECONDARY 


TRAINING SYSTEMS 


* Certificate and degree programs 
offered by local colleges and 


universities 


CONTEXTUAL 


LOCAL ECONOMY OTHER COMMUNITY FACTORS 


+ Job openings in demand * Size, characteristics of target 
occupations population 

+Industry growth in demand * Other service providers 
occupations + Referral partners 
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As shown in Exhibit 2-1, starting in the box at the left, the career pathways theory of change (as 
it applies to VIDA) begins with two types of program inputs:”° 


Organization. Organizational inputs include the lead agencies such as VIDA and its staff, 
funding from local governments and other organizations (e.g., foundations), and partner 
colleges that provide training. 


Participants. This individual input includes the characteristics of the target population. 
VIDA targets residents of one of the four counties in the Lower Rio Grande Valley who 
are 18 years or older, have at least a high school diploma or equivalent, be eligible to 
work in the United States, and have an interest in a career in which the program 
supports training. In addition, participants must be unemployed, underemployed, meet 
federal poverty income levels, or be on public assistance. 


This same box includes four kinds of program components expected to improve outcomes by 
helping participants overcome specific barriers that may make it harder to successfully enter 


into and complete occupational training: 


Assessment. VIDA uses placement sessions to determine eligibility based on applicants’ 
academic skill levels and whether they are committed to and capable of attending 
school full time in one of VIDA’s supported programs. For treatment group members, an 
additional meeting assesses financial needs and other potential barriers the program 
will help participants address. 


Instruction. This includes the College Prep Academy and the college occupational 
training programs that VIDA supports. Importantly, control group members can access 
the same occupational training coursework (but not the College Prep Academy), so 
particular features of these courses do not differentiate the experiences of the two 
groups. 


Supports. VIDA requires weekly advising and case management sessions, and provides a 
broad array of financial supports. 


Employment. VIDA provides pre-employment preparation and some assistance to 
program completers looking for work, but its most prominent employment activity is 
staying up-to-date on labor market trends so that it supports only training for 
occupations in high demand in the Lower Rio Grande Valley. 


The middle box shows the intermediate outcomes, where improvements are expected to lead 
to better main outcomes. These intermediate outcomes include improved basic skills for 


20 


Program inputs can include both components available only to treatment group members and components 


available to treatment and control group members, because the interaction of the former components with 
the latter can lead to impacts. 
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participants who enroll in the College Prep Academy and improved psycho-social skills such as 
grit and academic self-confidence; attainment of occupational-specific skills; career knowledge 
and college success skills; and reduced financial hardship. 


In the far right box, the main outcomes are the primary targets that programs seek to change. 
For VIDA, these include: 


e Increased postsecondary attainment, namely accumulated college-level credits and 
occupational training certificates and degrees; 


e Successful employment in middle-wage jobs or higher, increasing earnings and job 
benefits, and career advancement; and 


e Improvements in other outcomes, such as individual and family well-being. 


A number of contextual factors may influence expected effects. These include the colleges in 
the local area; the local economy and other community factors, such as the size and 
characteristics of the target population; and the number and nature of service providers. 


2.2 Research Questions for the Evaluation of VIDA 


The implementation study documents VIDA as implemented and captures participation 
patterns of treatment group members in training and other activities (see Chapter 4 for its 
findings). The impact study (see Chapter 5) aims to measure the effectiveness of VIDA in 
improving students’ intermediate and main outcomes. 


Implementation study questions: 


e What is the intended program model? What is its institutional and community context? 


e What intervention was actually implemented? Did it deviate from plans or 
expectations? 


e What were the treatment group’s participation patterns and experiences with program 
services? 


e What are the differences in services, including training, received by treatment and 
control group members? 


Impact evaluation questions: 


e What were the main effects of VIDA on: 


— Educational attainment, including college credits received, credentials received, and 
other educational outcomes? 


— Entry into career-track employment, higher-wage jobs, earnings, and perceptions of 
career progress? 
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— Participant and family well-being, including income and material hardship? 


e To what degree did the program affect intermediate outcomes in the theory of change, 
such as: 


— Confidence in career knowledge and access to career supports? 


—  Psycho-social skills such as grit, academic self-confidence, core self-evaluation, and 
social belonging at school? 


— Life stressors, such as financial hardship, life challenges, and perceived stress? 


As noted above, the program’s theory of change not only describes the hypothesized causal 
relationships between inputs and outputs, it also identifies time horizons over which effects are 
expected to occur. For example, because some VIDA participants will first enter the College 
Prep Academy, others will be newly enrolling in college, and still others will already be enrolled 
in an occupational program, the program expects that only a subset of treatment and control 
group members will have had time to complete their programs during the 24-month follow-up 
period. Thus, this early impact report primarily focuses on outcomes such as college enrollment 
and credits earned, neither of which require that a participant has completed a program and 
earned a credential. 


Later PACE reports will focus more on employment outcomes, such as earnings and career 
advancement, and on education and training outcomes (including credentials) that a substantial 
proportion of sample members may require a longer time to complete. 


2.3. PACE Evaluation Design and Analysis 


As mentioned in Chapter 1, the PACE evaluation uses a random assignment research design to 
estimate the impact of access to the program on students’ outcomes. The great benefit of such 
a design is that when properly implemented, it ensures that estimated effects reliably can be 
attributed to access to the program and not to unmeasured differences in characteristics or 
external circumstances between individual students with access (treatment group) and without 
access (control group) to the program. 


The evaluation estimates the impact of the opportunity to access the program, rather than of 
actual receipt of any of its components. Evaluators refer to such analyses as “intent to treat.” 
With only one treatment group, this form or analysis provides rigorous estimates only of the 
overall effects of the program offer—it does not provide experimental estimates of the receipt 
of particular program components.~* 


21 In aresearch design with multiple treatment groups some of which lacked one or more program components, 


it is possible with an ITT analysis to estimate the effects of a particular program component, but even in sucha 
case, this would estimate the effects of access to the component, not its actual receipt. 
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A second feature of the impact study design is that both treatment and control group members 
can access education, training, and support services available in the community that are not 
exclusive to the program PACE is evaluating. That is, the evaluation estimates the effect of the 
program’s components above and beyond what was otherwise available at colleges in its 
catchment area and elsewhere in the community during the study period. For example, for 
VIDA, both the treatment and control group members could access the same college 
occupational training courses. Thus, the control group’s experiences represent what would 
have happened absent VIDA’s program components. 


In summary, the PACE impact study assesses whether the existence of this multi-component 
career pathways program led to better outcomes for students who were offered the chance to 
participate in VIDA, given what these students could have obtained without the program.” 


2.3.1 Intake and Random Assignment Procedures 


The research team worked closely with each program in the PACE evaluation to design and 
implement program intake and random assignment procedures. The steps in VIDA’s procedures 
during the study were as follows: 


e Recruitment. VIDA received funding from the PACE evaluation for enhanced 
recruitment in order to effectively quadruple the number of applicants to the program 
in order to double enrollment in the program and to establish a control group equal in 
size to the treatment group for the study.” To meet its recruitment goals, during the 
period of the study, VIDA extensively marketed its program through media, partners and 
word-of-mouth referrals. Partner colleges often referred enrolled students with 
financial problems. 


Interested potential applicants who wanted further information about the program 
before applying could attend an optional monthly orientation session to learn about 
VIDA services, its eligibility and participation requirements, how random assignment 
governed admission to it, and the certificate or degree programs VIDA supported. There, 
they could sign up to attend a group placement session. Typically lasting a large part of 


*2 Four technical appendices provide additional details about analysis methods. Appendix A describes data 


collected at baseline, gives further detail on baseline characteristics of treatment and control group members, 
and explains procedures for using these data to adjust for imbalances arising by chance during random 
assignment. Appendix B describes college records data serving as the main source for measuring program 
impacts on educational progress. Appendix C provides detail on survey-based outcome measures, adjustments 
for item non-response, and analyses of survey non-response. Finally, Appendix D documents the research 


team’s approach to outliers. 


73 In order to double the number enrolled in VIDA, the program received funds from an Open Society 


Foundations grant to Abt Associates, as well as directly from the Kresge, Laura and John Arnold, Meadows and 
Hearst Foundations. 
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the day, the placement sessions occurred weekly, rotating to different cities and towns 
across the four-county region. 


Eligibility. At the placement session, VIDA staff assessed applicants’ eligibility based on 
education level, age, employment status, citizenship, and ability to commit to full-time 
education and training. 


Determination of eligibility and suitability for VIDA: Applicants completed VIDA’s 
application form and then met individually with a VIDA Counselor to determine their 
eligibility for the program. To be eligible for VIDA, they had to be unemployed, 
underemployed, meet federal poverty income levels, or be on public assistance. 
Additionally, they had to be residents of the Rio Grande Valley, 18 years or older with a 
high school diploma or GED, and eligible to work in the United States.”* The Counselor 
discussed the applicant’s educational background, training interests, career goals, and 
potential barriers to full-time participation in education and VIDA services. The 
Counselor also confirmed the applicant’s education and employment interests in one of 
the programs of study that VIDA supported. 


Education assessment: Some applicants were not college ready because they could not 
pass the college entrance exams. VIDA assessed applicants during the placement 
session, requiring a minimum education level of at least a tenth grade proficiency level 
in math, reading, and writing to be eligible for College Prep Academy. College-ready 
applicants who had not yet applied to college were required to take the college 
entrance exam prior to the placement session and bring their scores to it. Those already 
enrolled in college presented copies of their transcripts and college entrance exam 
scores. 


As discussed above, VIDA required that participants commit to attending school full time and 
participating in mandatory, weekly counseling sessions. Following their meeting with the 


Counselor, applicants met with a senior staff member who made the final decision about 
whether they would proceed with the VIDA and PACE study enrollment process. Random 
assignment occurred at the conclusion of the placement session. 


Informed Consent. Eligible applicants were asked to sign the PACE evaluation informed 
consent form in order to enroll in the program. Applicants who refused to sign the 
informed consent form were not included in the study and were not eligible for VIDA 
services. Those who signed the form became study participants. 


Baseline Data. VIDA applicants completed the PACE Basic Information Form (BIF), which 
the program had incorporated into its application form, and the PACE Self-Administered 


24 


VIDA referred applicants who did not have a high school diploma or GED to local GED training programs, such 


as those offered by American Job Centers. 
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Questionnaire (SAQ). The BIF collected demographic and economic information. The 
SAQ measured a variety of attitudes, beliefs, and psycho-social dispositions, as well as 
more sensitive and personal characteristics. 


e Random Assignment. Program staff used an online system to randomly assign study 
participants to the program or control group. The random assignment ratio was 1:1, so 
that the treatment and control groups would each include approximately half of the 
research sample. 


e Services According to Random Assignment Status. Study participants assigned to the 
treatment group had access to VIDA services (but were not required to use them). 
Before leaving the placement session, treatment group members scheduled an intake 
appointment with the Counselor at VIDA’s office and received a career assessment to 
complete beforehand and bring to the appointment. Those assigned to the control 
group could not access the VIDA services but could access other services available in the 
community, including at VIDA’s college partners. 


Between November 2011 and June 2014, VIDA staff randomly assigned 958 study participants: 
478 to the treatment group and 480 to the control group. 


2.3.2 Characteristics of the Study Sample 


Exhibit 2-2 (below) shows the percentage distributions of the treatment and control group 
members across a series of characteristics. The p-values in the last column test the hypotheses 
that there is no systematic difference between the groups for each characteristic.” As shown, 
random assignment produced treatment and control groups without significant differences in 
observed baseline characteristics with two exceptions: treatment group members were 
somewhat more likely than control group members to have college experience and to be 
working 20 hours or more. These differences are most likely due to chance, given the number 
of characteristics tested (see Hypothesis Testing section below). In conducting impact analyses, 
the research team controls for any bias resulting from these and other differences by using 
baseline values as covariates to adjust for chance differences (described in the Impact 
Estimation section below). 


Exhibit 2-2 also shows the composition of the study sample. The study sample’s characteristics 
are consistent with the nontraditional student population VIDA seeks to serve. Sample 
members are low income; approximately half have annual household incomes of less than 
$15,000, and more than 85 percent have incomes less than $30,000. Consistent with these low 


> The p-value summarizes results from chi-squared tests of the likelihood that the difference in the observed 


value or larger would occur if there were no differences between the two samples. For example, a p-value of 
.32 means that even if the characteristics of the members in the treatment and control groups were identical, 
the observed difference or larger would occur 32 percent of the time. 
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levels of income, about two-thirds received benefits from the Supplemental Nutrition 
Assistance Program (SNAP, formerly known as Food Stamps) or Special Supplemental Nutrition 
Program for Women, Infants, and Children (WIC). About two-thirds also reported experiencing 
financial hardship in the past year. Study participants are also older than traditional college 
students. More than 60 percent were age 25 and older, and more than one-fifth were age 35 or 
older. However, because VIDA required at least a high school diploma or GED and enrolled 
participants with prior college experience, their levels of education were relatively high; 
virtually all had at least a high school degree or equivalent, and more than half had a year or 
more of college. Although not captured in the BIF, other qualitative study evidence (see 
Section 3.3.1) indicates that this high level of prior education in part results from many study 
participants already being enrolled in college programs when they applied to VIDA. 


The majority of study participants (about 70 percent) were female. Nineteen out of 20 were 
Hispanic, reflecting the population of the Lower Rio Grande Valley, which is about 90 percent 
Hispanic. Most were not working at the time of random assignment, reflecting their interest in 
and ability to attend school full time. 
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Exhibit 2-2. Selected Characteristics of the VIDA Study Sample 
All Study Treatment Control Group 
Characteristic Participants (%) Group (%) (%) p-Value 
Age 222 
20 or under 14.1 12.3 15.8 
21 to 24 22.9 24.3 21.5 
25 to 34 40.6 39.3 41.9 
35 or older 22.4 24.1 20.8 
Sex 410 
Female 70.9 69.7 72.1 
Male 29.1 30.3 27.9 
Race/Ethnicity 235 
Hispanic 95.8 95.6 96.0 
Black Non-Hispanic 0.9 0.4 1,3 
White Non-Hispanic 3.0 3.6 2.4 
Other Non-Hispanic 0.1 0.2 0.0 
Current Education .038 
Less Than a High School Degree 0.7 0.2 1.3 
High School or Equivalent 26.1 23.0 29.2 
Less Than 1 Year of College 15.8 17.0 14.6 
1 or More Years of College 52.7 55.7 49.7 
Associate’s Degree or Higher 4.7 4.2 bid 
Income 238 
Less than $15,000 50.9 49.6 52.3 
$15,000 to $29,999 36.5 39.0 34.0 
$30,000 or More 12.6 11.5 13.8 
Mean $16,376 $16,277 $16,474 813 
Public Assistance / Hardship in Past 12 Months 
Received WIC or SNAP 67.6 66.5 68.8 442 
Received Public Assistance or Welfare 5.5 6.0 5.0 501 
Reported Financial Hardship 67.2 66.5 67.9 .630 
Current Work Hours Per Week 017 
0 64.9 62.8 67.0 
1to 19 11.8 10.9 12.8 
20 to 34 14.8 18.4 11.1 
35 or more 8.5 7.9 9.1 
Expected Work Hours Per Week in Next Few Months 321 
0 55.3 53.7 56.8 
1to 19 12.6 11.5 13.6 
20 to 34 21.0 23.3 18.8 
35 or more 11.2 11.5 10.8 


SOURCE: PACE Basic Information Form. 

SNAP is Supplemental Nutrition Assistance Program. WIC is Special Supplemental Nutrition Program for Women, Infants, and 
Children. 

NOTE: There are two significant differences at the p=.05 level. Appendix A provides a fuller set of baseline characteristics, also 
confirming that random assignment generated well-balanced treatment and control groups. Some percentages for characteristics do 
not add up to 100.0 percent due to rounding. Public Assistance/Hardship in Past 12 Months does not because the categories are 
not mutually exclusive nor exhaustive. 
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2.3.3. Analysis Plan for the Impact Study 


Prior to estimating VIDA impacts, the research team published an analysis plan specifying key 
hypotheses and outcome measures.”° The team subsequently assessed data quality, refined the 
plan, and publicly registered it on the Open Science Framework and the What Works 
Clearinghouse websites. 7 The purpose of the analysis plan and registration was to guide the 
work of the research team and publicly commit to particular hypotheses and an estimation 
approach and aligns with ACF’s commitment to promote rigor, relevance, transparency, 
independence, and ethics in the conduct of evaluations.”® 


Hypothesis Testing 

An essential principle in the PACE analysis plan is to organize and discipline the number of 
statistical tests conducted. Like most social policy evaluations, the nine PACE studies target an 
array of different outcomes. If the evaluation did not adjust in some way for multiple 
hypothesis tests, a potentially large number of the tests would reach the evaluation’s standard 
for statistical significance by chance, even if there were no actual effect on these outcomes. 
This is known as the problem of “multiple comparisons.” To address this issue, the team 
established three categories of hypotheses: confirmatory, secondary, and exploratory: 


e Confirmatory tests involve outcomes most critical to judging whether the program 
seems to be on track—that is, producing the results expected at a given follow-up 
duration. Given the relatively small sample sizes in the PACE studies, the research team 
generally limited such tests to one per program in the early impact report (at 20- 

24 months after randomization for VIDA) and two tests in each subsequent report (at 
three and six years after random assignment)—selecting outcome(s) under the “main” 
category in the program’s theory of change (see Exhibit 2-1). Each confirmatory 
hypothesis has an expected direction of change, an increase or decrease in the 
outcome. Therefore, the research team tests each confirmatory hypothesis for 
significance only in the specified direction, ignoring possible effects in the other, by 
applying a one-tailed test of statistical significance. The primary outcome in the VIDA 
early analyses is college credits. 


e Secondary hypotheses involve a set of additional indicators consistent with expected 
effects within the period covered by the study report. As with the confirmatory 
hypothesis, each secondary hypothesis has an expected direction of change, and 
therefore, the research team tests each secondary hypothesis for significance only in 
the specified direction by applying a one-tailed test of statistical significance. Secondary 

6 See Abt Associates, Inc. (2015). 


The Open Science Framework website is: https://osf.io/bpm8u/. The What Works Clearinghouse website is 
not currently online. 


27 


28 ‘ é 
See https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy. 
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analyses for VIDA included tests of hypotheses for additional education outcomes, as 
well as a number of indicators of early career progress. 


e Exploratory hypotheses cover an additional set of possible effects whose direction and 
timing are less certain. Accordingly, the team applies two-tailed tests to these 
hypotheses. Exploratory hypotheses for VIDA looked at psycho-social skills 
(e.g., academic self-confidence, social belonging in school) and life stressors 
(e.g., financial hardship, perceived stress). 


Chapter 5 identifies the specific hypotheses in each category tested for VIDA. 


Impact Estimation 

Random assignment ensures that, on average, samples of treatment and control group 
members will have similar characteristics at the outset. As a result, measured differences in 
outcomes provide unbiased estimates of program impacts. Following common conventions, we 
compensate for chance differences in measured baseline characteristics when estimating 
program outcomes. Doing so also helps to increase the precision of estimates. 


For PACE, the research team estimates a statistical model relating each outcome to baseline 
variables for the control group sample, then calculates average differences between actual and 
predicted values in both groups and subtracts the control group average from the treatment 
group average to generate the impact estimate. Appendix A provides a detailed description of 
this method.” 


We estimated this approach both for continuous outcomes (e.g., total college credits earned) 
and for binary outcomes (e.g., yes/no questions). For survey-reported outcomes, weights were 
used to average outcomes. Additional details can be found in the technical appendices. 


Formally, estimation uses the following equation: 


¥1(%-¥)-—d0-7)(%-¥), 
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where 6 is the estimated impact of being in the treatment group (whether or not the person 
attended the program or used any of the offered services); Y is the observed outcome of 
interest (e.g., credits); Y is a prediction of Y based on baseline variables measured at random 
assignment; T is an indicator of treatment status (which is set equal to 1 if the individual is 
assigned to the treatment group and 0 if the individual is assigned to the control group); n,and 


29 As explained in the appendix, the approach is a variant on the traditional approach to regression-adjustment 


methods used in impact analyses. The latter typically involves linear regression of each outcome on an 
indicator of treatment status and a series of baseline variables. In this approach, the coefficient on the 
treatment indicator provides the regression-adjusted impact estimate. 
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n-are the respective sample sizes in the treatment and control groups; and the subscript / 
indexes individuals. 


2.3.4 Analysis Plan for the Implementation Study 


The PACE evaluation’s implementation study relies on both qualitative and quantitative 
analyses, using a broad variety of data sources. Key analyses include the following: 


2.4 


Qualitative description of implementation. Describing the program’s design and 
context and developing its theory of change relied primarily on review of program 
materials (e.g., its scheduled topics for weekly advising sessions, in the case of VIDA); in- 
person discussions with program staff and leadership during two rounds of site visits; 
and biweekly or monthly calls between study and program leadership during the study 
period when random assignment was ongoing. 


Quantitative analyses. A quantitative analysis of the proportion of program participants 
whose educational experience conformed to VIDA’s program goals (e.g., they 
maintained full-time enrollment in college) and who reached major program milestones 
serves to systematically document their experience in the program. This relies on 
college records, follow-up surveys of treatment and control group members, and in the 
case of VIDA, its financial information system, which contains information on all 
payments on behalf of each treatment group member. 


Fidelity. VIDA’s financial information system also provides data on the extent to which 
program participants received substantial financial supports. In addition, the study 
relied primarily on field observations and interviews with program staff and participants 
to assess the delivery of VIDA’s counseling services. To address the question of how 
program delivery changed over time, the research team asked program staff about 
internal or external obstacles and how staff altered the program in an attempt to 
overcome them. 


Service Differences. The random assignment design of the impact study implicitly 
assumes that any effects of the program on its main outcomes result from differences in 
the programmatic experiences of treatment and control group members. Thus, where 
possible, the implementation study describes the difference in services the two groups 
received. This is important for the PACE evaluation, as the control group is not barred 
from receiving services available in the community, including those similar to what the 
treatment group can access through the program. 


Data Sources 


The PACE evaluation’s implementation and impact studies use a variety of data sources. 


Baseline Surveys. Prior to random assignment, study participants completed two 
baseline surveys: The BIF collected demographic and economic information, and the 
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SAQ) measured a variety of attitudes, beliefs, and psycho-social dispositions, as well as 
more sensitive and personal characteristics. For the VIDA study, applicants who 
consented to participate in the study completed the BIF (which was incorporated into 
the program application) and SAQ at the placement session. 


Follow-Up Survey. The research team sought to survey all PACE study sample members 
starting at 15 months after random assignment. On average, for VIDA, the survey 
occurred 20 months after random assignment. The survey asked questions on study 
participants’ training and service receipt, postsecondary educational attainment, 
employment, income, debt, and participation in income support programs. It used a 
mixed-mode approach, conducted initially by telephone and then in person for those 
not reached by telephone. For the VIDA study, Abt’s survey unit, Abt SRBI, completed 
surveys with 434 treatment and 373 control group members, yielding response rates of 
91 percent and 78 percent, respectively. °° 


Administrative Records. The PACE team relied on the administrative records of each 
program evaluated, both to describe the experience of treatment group members in 
their program and to estimate program effects. For the VIDA study, the team used 
records from the five local colleges with which VIDA partners to measure treatment and 
control group members’ enrollment in education and training, as well as their credit and 
credential receipt. Analyses of wider National Student Clearinghouse (NSC) data for the 
sample revealed that 99 and 98 percent of treatment and control group members 
attending college after random assignment, respectively, attended these five and no 
other NSC-covered colleges.** The team also used VIDA’s case management data to 
analyze enrollment status, participation in the College Prep Academy, and program of 
study pursued. In addition, the team relied on VIDA’s accounting data to analyze the 
amount and pattern of participant receipt of financial assistance. 


Site Visits. For the implementation study, the evaluation team conducted two site visits 
to each PACE program. For VIDA, the first visit occurred in June 2012, about seven 
months after random assignment began. The goal of this visit was to document the 
program’s theory of change and key components and to assess implementation of 
evaluation procedures. The second visit was in April 2014, near the conclusion of 
random assignment. The visit documented any modifications to operations or the 
provision of services, as well as implementation challenges. During the visits, the 
research team interviewed VIDA Board members and program managers; staff involved 
in evaluation activities (e.g., recruitment, intake, random assignment); VIDA Counselors; 


See Appendix C for response bias analyses. 


The National Student Clearinghouse contains information on enrollment, degree attainment and other 
educational outcomes from 3,600 colleges and universities. 
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local college administrators and College Prep Academy instructors; and representatives 
of local VIDA funders. 


e In-Depth Interviews with Sample Members. The research team conducted in-depth 
interviews with a random sample of treatment and control group members. A 
researcher traveled to the Lower Rio Grande Valley in May 2014 and conducted 
interviews in person. Interviews lasted on average 45 minutes. The team interviewed 
12 study participants for this report (eight treatment group members and four control 
group members). The interviews focused on reasons for wanting to enroll in VIDA, 
desired outcomes, participation in the program (treatment group only), other services 
received (control group only), and perceived facilitators and barriers to education and 
training and career success. 


e Program Documents. The research team obtained and reviewed program documents, 
including annual reports; program materials such as applications, education and career 
assessments, financial worksheets, and counseling materials; and college course 
catalogs. 
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It is important to understand the context in which VIDA operates and its program design to 
properly interpret implementation and impact findings. This chapter begins with a description 
of local context during the PACE random assignment period (2011-2014). It then discusses 
program administration in the context of VIDA’s history as an organization. Finally, the chapter 
describes VIDA’s program design and the delivery of services to its participants. >” 


3.1 Local Context 


Three aspects of the local environment are important to understanding VIDA’s design, 
implementation, and impacts: the characteristics of the population in the region VIDA serves, 
the local labor market, and the services and opportunities available to control group members. 


3.1.1 Population 


As noted, VIDA enrolls adults who are unemployed, underemployed, meet federal poverty 
income levels, or are on public assistance. All have to be at least 18 years of age, legal residents 
or U.S. citizens, have a high school diploma or GED, and have at least tenth grade level math 
and literacy proficiency. VIDA serves a four-county region in the Lower Rio Grande Valley of 
Texas. Cameron, Hidalgo, Starr, and Willacy Counties together are 43,000 square miles, and 
they had a combined population of 1.3 million in 2014. The two major cities in the region are 
Brownsville (in Cameron County) and McAllen (in Hidalgo County). During the PACE study, 
VIDA’s headquarters were in Weslaco, also in Hidalgo County. As shown in Exhibit 3-1 (below), 
Cameron and Hidalgo Counties were more populous (415,000 and 806,000, respectively) than 
Starr (62,000) and Willacy (22,000). 


In 2014, the population in all four counties was of predominantly Hispanic or Latino ethnicity 
(88 to 96 percent), specifically Mexican (85 to 94 percent). Among those who were not Hispanic 
or Latino, the largest share was White (4 to 13 percent). Compared with the U.S. as a whole, the 
population in the Lower Rio Grande Valley had lower household incomes and a greater 
proportion lived below the federal poverty level. Among the four counties served by VIDA, 
these disparities were even more notable in Starr and Willacy. The median household income 
nationally was $53,000 in 2014, whereas the income levels in Starr and Willacy Counties were 
about half that ($26,000 and $28,000, respectively). Those in Cameron ($34,000) and Hidalgo 
($35,000) were still well below the national average but somewhat higher than elsewhere in 
the region. About 16 percent of the U.S. population lived below the federal poverty level in 
2014, compared with more than one-third of the population in each of the four counties VIDA 
serves. 


2 For additional information about the program see Copson, Gardiner, and Rolston (2014). 


Abt Associates 3. About VIDA: Context and Program Design 1 pg. 23 


Valley Initiative for Development and Advancement 
Implementation and Early Impact Report 
PACE 


These counties also experienced a higher unemployment rate than the national average. When 
PACE study enrollment began in 2011, the national unemployment rate was nine percent; in 
Cameron and Hidalgo Counties it was 12 percent, and in Starr and Willacy Counties it was 

17 percent. When PACE enrollment concluded in 2014, the national unemployment rate had 
declined to five percent, compared with seven percent unemployment in Cameron and Hidalgo 
Counties and 12 and 14 percent in Starr and Willacy Counties, respectively (U.S. Bureau of 
Labor Statistics, “Local Area Unemployment Statistics”). 


The population in VIDA’s service area also had lower educational attainment than the national 
population. In Starr and Willacy Counties, more than half of the population had not attained a 
high school diploma compared with about one-third of the populations in Cameron and Hidalgo 
Counties and 14 percent of the population nationally. About one-fifth of the population in the 
four counties had a high school diploma or GED. A greater proportion (one-fifth) of Cameron 
and Hidalgo County residents had completed an associate’s, bachelor’s, or graduate degree, 
versus 13 and 10 percent, respectively, of Starr and Willacy County residents. 


Exhibit 3-1. Characteristics of the Program Environment for VIDA, 2014 


United Cameron _— Hidalgo Starr Willacy 
States County County County County 
Total Population 314,107,084 415,103 806,447 62,040 22,056 
Race and Ethnicity (%) 
Hispanic or Latino of any race 16.9 88.4 90.9 95.7 87.4 
Mexican 10.8 85.6 88.4 93.9 85.2 
Other Hispanic or Latino 6 2.9 2.5 1.7 2.2 
Not Hispanic or Latino 83.1 11.6 9.1 4.3 12.6 
White alone 62.8 10.3 74 3.4 10.5 
Black or African American alone 12.2 0.4 0.4 0 1.6 
Some other race alone 6 0.8 1.1 0.5 0.3 
Two or more races 2.1. 0.2 0.2 0.4 0.2 
Educational Attainment @ (%) 
No High School Diploma 13.6 35.9 37.8 53.3 53.2 
High School Graduate (Includes Equivalency) 28 24 23.7 21.8 22.5 
Some College, No Degree 21.2 18.5 17.5 12.5 13.9 
Associate’s Degree 7.9 5.6 4.6 2.9 2.3 
Bachelor's Degree 18.3 11.5 11.8 71 5.7 
Graduate or Professional Degree 11 4.6 4.7 2.5 2.4 
Median Household Income ($) 53,482 33,390 34,952 25,906 27,627 
Percentage below Poverty Level » 15.6 34.8 34.6 38.9 38 
Unemployment Rate (%) 5.3 71 7.2 13.6 12.4 


SOURCES: 2014 data as reported by the American Community Survey 2010-2014. 

* Among those age 25 and over. 

» Among the entire population; as defined by the U.S. Census Bureau, accessed November 21, 2016, at 
http://www.census.gov/acs/www/data_documentation/documentation_main/#doc2012. 
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3.1.2 Local Labor Market 


A second contextual factor is the size of the local labor market with respect to jobs in the 
occupations for which program participants trained. VIDA’s model entails that program 
leadership remain abreast of economic and labor market trends in the Lower Rio Grande Valley 
so that the training supported by the program aligns with current and anticipated employer 
needs. VIDA partners with local economic development corporations (EDCs) to learn about 
existing businesses in the area with job openings in certain occupations and about companies 
or industries the EDCs seek to attract to the area and what skills and competencies those 
employers would require of employees. VIDA uses this information to determine the programs 
of study in which it will support participants’ training. VIDA periodically updates its list of 
supported programs to reflect labor demand changes reported by the EDCs. 


According to May 2013 occupational employment data from the U.S. Bureau of Labor Statistics, 
the top five major occupational groups in the McAllen, Edinburg, and Mission region (Hidalgo 
County) were office and administrative support (16.6 percent of local employment); sales and 
related (12.6 percent); personal care and service (11 percent); education, training, and library 
(10.4 percent); and food preparation and serving (8.8 percent). Healthcare was another sector 
employing a significant share of the population, in healthcare practitioner and technical 
positions such as Physician and Surgeon, Registered Nurse, Medical and Clinical Laboratory 
Technician, and Paramedic (6.1 percent) and in healthcare support occupations such as Home 
Health Aide, Medical Assistant, and Nursing Assistant (4 percent) (U.S. Bureau of Labor 
Statistics, “Occupational Employment and Wages in McAllen-Edinburg-Mission”). 


In study interviews, EDC staff emphasized the importance of the healthcare sector in the local 
economy and the demand for skilled workers in this field. There are a large number of hospitals 
and outpatient care providers in the region. VIDA staff and Board members said that in the 
past, healthcare providers had to recruit talent from outside the region, but that the local 
colleges and universities were attempting to train local residents for those jobs. Situated along 
the U.S.-Mexico border and with a deep water seaport in Brownsville (on the Gulf of Mexico), 
the Lower Rio Grande Valley has also historically been a center of trade, production, and 
commerce between the two countries that drives jobs in manufacturing, transportation, and 
logistics. EDC staff anticipated growth in transportation and logistics. They also identified the oil 
industry as a potential source of growth, because of investments in shale extraction and 
offshore drilling, and aeronautic manufacturing and technology, as a result of SpaceX’s decision 
to construct a launch site near Brownsville. 


3.1.3 Control Group Environment 


The third contextual factor (and the one most pertinent to the evaluation’s random assignment 
design) is the degree to which program applicants randomly assigned to the study’s control 
group had access to education, training, and supports similar to the treatment group’s. Program 
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impacts can be more readily observed when the treatment group has available a package of 
services distinct from that available to the control group. 


Exhibit 3-2. Comparison of Career Pathways Program Components Available to PACE Control 


Group versus Treatment Group Members in VIDA 


DARS 

Career centers and employment 
services available from the colleges and 
universities 


Career 
Pathway Available to Treatment Group Only 
Component _ Available in Community to Both Groups (Additional Services in VIDA) 

Assessment e College entrance exams Individual assessment appointment completed with a 
VIDA Counselor after enrollment to review education and 
employment goals, barriers, and financial need 
Routine assessment of financial need throughout program 

Instruction Developmental education in math, College Prep Academy for instruction in math, reading, 

reading, and writing at area community and writing provided in condensed 16-week full-time 
colleges format at no cost 

Education and occupational training Targeting of education and occupational training programs 
programs at area community colleges at area community colleges and universities or other 

and universities or other training training providers by VIDA to in-demand occupations in 
providers the region 

Supports Tutoring and academic advising Dedicated VIDA Counselors 

through area community colleges Mandatory weekly group or individual counseling sessions 
Career and personal counseling Substantial tuition assistance aligned with each 
through local social service providers, participant's financial need 
including American Job Centers (AJCs) Financial assistance to pay for course-related books, 
and Texas Department of Assistive and tools, uniforms 
Rehabilitative services (DARS) For College Prep Academy participants, VIDA pays for 
Federal financial aid to attend school two rounds of college entrance exam testing 
Limited grants and scholarships to Financial assistance to pay for transportation and 
attend school childcare related to attending school 
Employment Services available from local social Identification of in-demand occupations in region designed 
Services service providers, including AJCs and to facilitate employment upon training completion 


Job search-related topics covered in group counseling 
sessions 

Phone call follow-up to graduates by program staff to 
determine employment status; offer of resume review if 
graduate is unemployed 


SOURCE: Program documents and site visits. 


Developmental education. Community colleges in the region offer developmental education 
for individuals needing math, reading, and writing instruction in order to pass the college 
entrance exams before applying to college. However, according to VIDA staff, these programs 
typically take several years to complete and often use up students’ limited financial aid 


resources, whereas VIDA’s College Prep Academy is 16 weeks, roughly a semester, of full-time 
enrollment and at no cost to participants. 


Education and training. Three community colleges and two universities in the region offer non- 
credit and for-credit courses and award certificates and degrees. Control group members have 
access to the same education and occupational training programs as treatment group 
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members. In fact, many VIDA applicants are already enrolled in a college program, but seek its 
financial assistance to help them complete their training. 


Counseling and advising. The colleges and universities provide educational support services 
such as tutoring and academic advising to their students, and some social service providers 
offer counseling on personal issues. Case management services through the local AJCs and 
Texas Department of Assistive and Rehabilitative Services (DARS) are not as extensive as VIDA’s. 


Financial assistance. For education expenses, treatment and control group members can access 
federal financial aid (e.g., Pell grants), and the colleges and universities have financial aid offices 
to assist with the process. Through the Texas Public Educational Grant Program (TPEG), 
students who apply for financial aid to attend a public college or university can receive grant 
assistance toward tuition from that institution. The institution has the discretion to determine 
recipients and award amounts.*° However, according to VIDA staff, these resources are seldom 
sufficient to cover an entire one- or two-year training program. Some colleges have resources 
available to help students; for instance, South Texas College offers scholarships and can help 
students pay for books, but they are limited. 


Employment services. The AJCs and DARS provide employment services, focusing on moving 
clients into employment as quickly as possible. The colleges and universities have career 
centers and employment services that both control and treatment group members could 
access. Services available through these centers include assistance with resumes and interview 
skills and access to job databases and online application tools. These are topics VIDA Counselors 
cover in weekly sessions. When necessary, VIDA refers participants to the career centers for 
additional assistance. 


In summary, the four counties served by VIDA in the Lower Rio Grande Valley had a large 
population of individuals who met VIDA’s target group profile and from which VIDA could 
recruit for its program. VIDA tailors its supported programs of study to local labor market 
demand according to information gathered from its economic development partners, and thus 
it is expected that jobs in the occupations for which participants train will be available upon 
their program completion. Finally, though treatment and control group members had access to 
the same college and university programs, the level of counseling and financial assistance 
provided by VIDA—as well as the packaging of them together—is unique to VIDA and not 
known to be available elsewhere in the region. 


3 Texas Higher Education Coordinating Board, “Texas Public Educational Grant Program (TPEG),” College for All 


Texans website, accessed January 6, 2017, 
http://www.collegeforalltexans.com/apps/financialaid/tofa2.cfm?ID=406. 
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3.2 Program Administration 


When the PACE study began, VIDA was an established organization with a history rooted in 
community organizing, alignment with economic development organizations in the region, and 
partnerships with the major educational institutions. This section describes VIDA’s history and 
goals that informed the design of the program and its implementation. It also discusses VIDA’s 
organizational structure, staffing, and partnerships. 


3.2.1 VIDA History and Goals 


The concept for VIDA began with Valley Interfaith, a community-organizing group in the Lower 
Rio Grande Valley affiliated with the national Industrial Areas Foundation (IAF). Valley Interfaith 
members observed a need among low-skilled and low-income adults in the region for education 
and training to secure good jobs with benefits and opportunities for advancement.“ 
Meanwhile, businesses and economic development boards looking to attract and retain 
employers in the region reported a need for a more highly skilled workforce. Valley Interfaith 
Saw an opportunity to meet the needs of both Valley residents and employers, and in 1995 
established a Board of Directors and hired an Executive Director to develop and implement a 
workforce training program. 


VIDA’s founders looked to Project QUEST, an organization in San Antonio with similar roots in 
community organizing, as a model for VIDA. Project QUEST provided funding for its participants 
to attend training, as well as counseling and support services to aid participants during their 
time in the program. VIDA adopted these elements, also adapting the model to the specific 
context in the Lower Rio Grande Valley and the interests of its Board. In particular, the Board 
was committed to focusing on education and training that resulted in an associate’s degree for 
several reasons: (1) employers indicated they needed employees with such degrees, (2) offering 
an associate’s distinguished VIDA from workforce development boards that funded short-term 
certificate training, and (3) the Board believed that individuals would more readily return to 
college for a bachelor’s degree at a later point if they had already achieved an associate’s 
degree. 


Currently, 10 programs operate a workforce development model similar to Project QUEST, 
adapted to the local context of their service areas.*’ Thus, when VIDA entered PACE, it had 


* In the late 1980s, according to VIDA staff, a number of textile manufacturers moved their production 


operations from Mexico to McAllen and employed a significant number of workers. However, by the mid- 
1990s, several large factories closed, resulting in job losses for a significant proportion of the population. 
Around the same time, the agriculture sector that also employed a large share of residents experienced a 
decline after an unusual freeze caused crops to fail. VIDA staff reported that the combination of these two 
factors contributed to an economic crisis in the region, as many households experienced a significant loss of 


income. 


a Project QUEST website, accessed December 19, 2016, at http://www.questsa.org/histor 
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15 years of operating experience with a model that had been replicated numerous times. That 
the model has proven to be capable of implementation in multiple sites makes this report’s 
findings of greater policy relevance than if they were only about a standalone model (Elliott and 
Roder 2017). VIDA’s goals for participants have not changed since its founding. The short-term 
goal, within one to two years after program entry, is for participants to graduate with an 
associate’s degree or industry-recognized certificate in a high-demand occupation and achieve 
employment in their area of study earning a living wage. Though VIDA’s founding Board 
emphasized associate’s degree programs, VIDA supports enrollment in certificate programs too, 
because they are viewed as a means to attract applicants who might otherwise be discouraged 
from the perceived rigor and time commitment of an associate’s program. VIDA expects that 
once enrolled, many of these certificate program participants will see that they can succeed ina 
college environment and might continue to an associate’s program or complementary 
certificate. Within certificate programs, VIDA typically supports only those programs that 
require a year or more, but it makes a limited number of exceptions. For example, some 
applicants have severe economic needs that make completing a longer program difficult, and 
some older workers seek skill development but prefer to return to work quickly. In addition, as 
described in later chapters, VIDA also supports some shorter certificate programs (e.g., Patient 
Care Assistant) as steps in longer credential programs (e.g., Licensed Vocational Nurse and 
Associate’s Degree in Nursing). 


Ultimately, VIDA’s goal is to change participants’ perceptions of and attitudes towards 
postsecondary education such that they see postsecondary education as a means out of 
poverty and as an attainable goal for themselves and for their family members and peers. For 
someone who earns an associate’s degree with VIDA’s support, that may mean they return to 
college for a bachelor’s degree; for parents, it may mean that they encourage their children to 
incorporate college or technical education into their future plans. 


3.2.2. Organizational Structure, Staffing, and Governance 


VIDA’s management team included five staff during the evaluation. *° An Executive Director had 
overall responsibility for carrying out the mission of the organization, fundraising, and 
supporting other staff. The Executive Director for most of the PACE period reported that an 
important quality to have in this leadership role is an understanding of business. There is both 
an organizational need to manage finances and funders, as well as a need to engage with the 
business community given VIDA’s emphasis on providing training relevant to current regional 
business needs. Board members stated that the Executive Director should have strong ties to 
the community to build and maintain necessary relationships and partnerships to support 
VIDA’s mission. 


© At the time of writing this report, VIDA had four staff on the management team and the position of Director of 


Program Services no longer existed. 
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The other four members of VIDA’s management team were: 


1. the Compliance Officer, who determined the level of financial assistance to be provided 
for each participant and identified appropriate sources of funding; 


2. the Director of Finance, who managed organizational finances, including assistance 
(e.g., tuition, support service payments) provided to VIDA participants; 


3. the Director of Program Services, who oversaw enrollment of participants, supervised 
VIDA’s Counselors, and contributed to fundraising and grant reporting; and 


4. the Director of Information Services and Community Outreach, who led recruitment 
efforts, managed organizational operations, and contributed to grant reporting. 


Individual counseling is a central component of VIDA’s program, and each participant has a 
Career Counselor-Case Manager (hereafter referred to as a “Counselor”). VIDA had six staff in 
this Counselor role, each with a caseload of approximately 50 participants. The Counselors are 
responsible for monitoring the academic progress of each participant, providing individualized 
support for their personal needs, holding weekly counseling sessions (either in groups or one- 
on-one), and ensuring that participants comply with program requirements. See Section 3.3.2 
for more information on VIDA’s counseling services. 


Several other staff perform support functions for the program. The Retention Specialist- 
Placement Coordinator’s role, which was created in 2012, is twofold. First, this staff member 
remains in regular contact with participants who have temporarily suspended their enrollment 
in training and attempts to reengage them. Second, the staff person contacts VIDA graduates 
every six months for two years to collect data on their employment status, wages, and benefits. 
Two Case Aides assist with data entry for program enrollment and for the Counselors. 


The Board of Directors is involved with organizational decision making and approves VIDA’s 
policies and budgets. Valley Interfaith continues to play a role in VIDA’s operations, with almost 
half of the 11-person Board having ties to Valley Interfaith. Other Board members include 
business people, non-profit leaders, and staff at local educational institutions. The region that 
VIDA serves is geographically large, so it makes efforts to include Board representation from all 
four counties. 


Partners and Funders 


In addition to partnering with higher education institutions in the Lower Rio Grande Valley to 
recruit and train VIDA participants, the program partners with local EDCs, cities and counties in 
the region, and regional workforce development boards to identify growing sectors of the 
economy where there is a need for skilled labor. VIDA receives funding from these entities to 
cover the costs of training VIDA participants in the occupations identified in consultation with 
these organizations as being in high-demand. During the PACE study, VIDA also received 
funding from the evaluation to put toward recruitment in order to effectively quadruple the 
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number of applicants to the program in order to double enrollment in the program and to 
establish a control group equal in size to the treatment group for the study. In order to double 
the number of participants enrolled in VIDA, the program received funds from an Open Society 
Foundations grant to Abt Associates, as well as directly from the Kresge, Laura and John Arnold, 
Meadows, and Hearst Foundations. 


EDCs, city governments, and county 
governments fund VIDA with the 
expectation that their dollars be spent on 
residents from their localities. The amount 


VIDA’s Primary Partners during PACE 


e Economic development corporations 
e Cities and counties 
e Workforce development boards and American Job 


of funding VIDA receives from each locality Centers 

influences where the organization focuses Community colleges: 

its recruitment and enrollment efforts. Not e South Texas College (STC), McAllen 

all localities in the four-county region fund e Texas Southmost College (TSC), Brownsville 

VIDA. In the absence of their funding, VIDA e Texas State Technical College (TSTC), Harlingen 
cannot recruit and serve individuals from Universities: 

those areas unless it receives funds from e University of Texas at Brownsville (UTB), Brownsville* 


other sources that allow the program the e University of Texas Pan American (UTPA), Edinburg* 


flexibility to enroll individuals regardless of 
where they reside. Foundation and 
corporate grant funds typically allow VIDA 
this greater flexibility to enroll participants from outside locally-funded areas. At the participant 
level, VIDA often blends sources of funds for participant tuition and financial assistance, 
meaning that a given individual’s participation in the program might be funded by an EDC or 
city and also by foundation dollars. According to staff, “braiding” resources in this way enables 
VIDA to distribute its restricted and unrestricted dollars more evenly and ultimately serve more 
participants. 


* Dissolved to form University of Texas 
Rio Grande Valley (UTRGV), Brownsville 


Economic development corporations (EDCs). Funded by local sales tax revenue, non-profit 
EDCs address job creation, education, and training in their jurisdictions. EDCs provide VIDA with 
information on the local labor market, and VIDA tailors its supported programs of study to align 
with current and expected job demand. In cases where a business is expanding or opening but 
its jobs are not yet available, VIDA often offers training in a related in-demand occupation to 
avoid preparing participants for specific positions that may not materialize. For example, if an 
EDC was recruiting an aviation maintenance company to the area, VIDA might propose to train 
participants in a mechanical trades program that would give them skills applicable to a variety 
of sectors, not just to aviation. In addition to information sharing, the EDCs also fund VIDA to 
serve residents in their jurisdictions. VIDA makes an annual appeal to each EDC to request 
funding, and the EDCs have the discretion to adjust their funding amount each year. During 
PACE, nine EDCs provided financial support to VIDA. 
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Cities and counties. A number of local city and county governments provide funding to VIDA to 
train their residents. VIDA’s leaders present annually to the cities and counties to update them 
on enrollment and outcomes for their residents who participate in the program. This update 
also includes an appeal for renewed funding for the next year. During PACE, two cities (Combes 
and Weslaco) and two counties (Hidalgo and Willacy) provided financial support to VIDA. 


Workforce development boards. Local workforce development boards provide online and in- 
person employment and training services to jobseekers and business services to employers. 
These organizations operate AJCs that offer job search and employment-related workshops, 
counseling, and resources. VIDA works with two local workforce boards: Workforce Solutions 
Cameron and Workforce Solutions Lower Rio Grande Valley. The AJCs refer potential applicants 
to VIDA, and VIDA refers prospective applicants who do not have their GED to the AJCs that 
offer GED training. 


Educational institutions. VIDA partnered with five institutions of higher education in the 
region. In addition to serving VIDA participants in their usual academic programs and services, 
these institutions also referred prospective applicants to VIDA and provided space on campus 
for VIDA Counselors to meet with participants. Three were community colleges that offered 
one-year certificates and two-year associate’s degrees: South Texas College (McAllen), Texas 
Southmost College (Brownsville), and Texas State Technical College (Harlingen).°” Two were 
universities within the University of Texas system—University of Texas Brownsville 
(Brownsville) and University of Texas Pan American (Edinburg)—that offered four-year 
bachelor’s degrees, as well as master’s and doctoral degrees. South Texas College was the 
longest-standing college partner, having been involved with VIDA since its early years. It 
partnered with VIDA to offer the College Prep Academy, along with Texas State Technical 
College. 


Late in the PACE enrollment period, several changes occurred among these educational 
providers. First, Texas Southmost College and the University of Texas Brownsville, which had 
been jointly affiliated for decades, terminated their relationship in Summer 2013 and began to 
operate as separate institutions following a transition period. VIDA’s Counselors reported that 
to minimize the impact of this change, they assisted participants who attended these 
institutions in navigating any uncertainty they had about course schedules or which institution 
would ultimately grant their degree. Second, the University of Texas system decided to dissolve 
the University of Texas Brownsville and the University of Texas Pan American and replace them 
with a single University of Texas Rio Grande Valley (UTRGV), with a main campus in Brownsville. 
The dissolution occurred in Summer 2015, and the first UTRGV class began in Fall 2015. 


37 South Texas College also offered bachelor’s degrees in Applied Science and Applied Technology and was one 


of only three community colleges in the state that had approval from the Texas Higher Education Coordinating 
Board to offer such degrees. 
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Although enrollment of participants into PACE concluded in Summer 2014, the follow-up period 
for the study continued through these transition periods. VIDA staff reported that participants 
were not substantially affected by the changes, but they nevertheless were relevant to the 
experiences of PACE study participants. 


3.3. Program Design 


The overarching theme of VIDA’s program design is high expectations with high support. As 
such, the program represents a mutual commitment between it and the participant: the 
participant commits to attending college full time, and the program commits to provide the 
financial and social supports necessary to enable credential attainment. This mutual 
commitment is reflected through all elements of VIDA’s design, from recruitment of 
participants through their obtaining employment in their chosen field of training. This section 
discusses recruitment strategies and the enrollment process for VIDA; support services and 
financial assistance available to treatment group members; program offerings available through 
VIDA including occupational education and training and the College Prep Academy; and 
employment services. 


3.3.1 Recruitment and Enrollment 


To quadruple its eligible applicant pool for the PACE study, VIDA substantially expanded its 
recruitment strategy. Prior to the study, VIDA did not actively recruit participants because the 
program maintained sufficient enrollment through word-of-mouth referrals, including from the 
local colleges and workforce development boards. Word-of-mouth referrals continued to be a 
major source for the program, according to staff. VIDA maintained its relationships with the 
local colleges and universities, and senior staff and Counselors routinely reminded their 
institutional contacts about the program and encouraged referrals. It was common for college 
staff to refer enrolled students encountering financial difficulties to VIDA. VIDA Counselors also 
posted fliers on campus with information about VIDA. In addition, word-of-mouth referrals 
came from current or former participants, their families and friends, and from workforce 
development boards. 


Throughout the study’s recruitment period, VIDA changed its recruitment strategy by adding 
public service announcements on public television stations in the region and advertising on the 
radio and in local newspapers. For a short time early in the PACE enrollment period, VIDA 
experimented with paid television advertisements, but found that these were no more effective 
than other strategies and were very costly. In addition to advertising the program in English, 
VIDA also advertised in Spanish and in Spanish-language media because of the large Spanish- 
speaking population in the region. 


The flexibility of its foundation funds also enabled VIDA to recruit and enroll participants in 
parts of the region where VIDA had previously lacked funding because the EDCs, city, or county 
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did not provide it. Staff expected that these areas would have considerable demand for VIDA 
services, because they had been underserved by the program in the past. 


As noted in Chapter 2, enrollment into VIDA’s program occurred during a group placement 
session held weekly, rotating among the cities and towns within VIDA’s service area. During the 
placement session, VIDA staff assessed applicants for eligibility, including discussing whether 
their education and job interests aligned with the programs of study supported by VIDA. Staff 
also determined applicants’ ability to attend school full time and participate fully in counseling 
sessions and other program requirements. VIDA developed the placement session format 
specifically for PACE in order to accommodate a greater number of applicants with its staff 
capacity. The session also included completion of a combined VIDA application and study BIF, 
the PACE informed consent form, and the PACE SAQ. Random assignment occurred at the 
conclusion of the placement session. Before leaving the session, treatment group members 
scheduled an intake appointment with the Counselor at VIDA’s office and received a career 
assessment to complete and bring to that appointment. 


3.3.2 Supports 


VIDA requires participants to maintain full-time enrollment in school®® and participate in weekly 


counseling sessions and periodic community service activities. VIDA designed its core support 
service components to aid participants to complete training and attain a certificate or degree 
applicable to their stated employment objective. Participants who find themselves unable to 
comply with its requirements are terminated from the program, unless they have a reasonable 
explanation or plan to address the issue. The upfront screening makes it likely that if it admits 
an applicant, VIDA could commit to meeting that applicant’s financial and support service 
needs—including education expenses—and that the applicant could be successful in the 
program. 


VIDA’s support service approach is highly proactive and intrusive for two reasons: to tailor 
financial assistance, such as for tuition, childcare, and transportation, to particular needs; and 
to ensure deep engagement with participants to prevent problems and deal with those that do 
arise despite detailed planning. This section discusses each core support component, 
illustrating its intrusive nature, as well as describing the program’s community service 
requirement. 


Individual Assessment 

Within a week of random assignment at the placement session, treatment group members 
meet individually with a Counselor at VIDA’s office for an intake appointment. Typically lasting 
about an hour, the meeting is designed to identify the individual participant’s career goals, 
educational path, and financial and support service needs. During the meeting, the Counselor 


38 - ‘i z . 
The study defines “full-time enrollment” as enrollment in classes bearing 12 credits or more. 
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reviews the career assessment completed by the participant after the placement session. VIDA 
uses the Harrington-O’Shea Career Decision-Making System, a brief booklet that participants fill 
out to help identify their job interests, values, and abilities. Counselors use it as the basis for a 
discussion about participants’ employment interests in relation to their skills and the nature of 
the jobs associated with the profession. 


Counselors reported that some participants already know what their employment and 
educational goals are, particularly those who are already enrolled in a college or university 
program. Other participants are uncertain of the exact job they want, or do not know how to 
attain it. For them, the Counselor discusses various options and prompts participants to 
consider whether the schedule, work environment, and associated training for the job align 
with their personal circumstances. For instance, if a participant is interested in nursing, but has 
a felony conviction that would make it difficult to get a job in that field, the Counselor suggests 
other occupations in healthcare without those restrictions. 


After determining an employment goal, the Counselor and participant develop a plan to 
identify where the participant is in the program of study (if already enrolled), what courses are 
required and a schedule for completing them, and the ultimate credential or degree awarded 
upon completion. 


An important activity of the intake appointment is to review participants’ finances and 
determine the amount of funding VIDA would commit to their training and support services. 
The Counselor reviews the participant’s income and expenses, public benefit receipt, and (for 
those already enrolled in school) any financial aid received. The Counselor also discusses 
potential barriers to participation—such as the need for childcare or transportation—that 
VIDA’s funds could help alleviate. VIDA provides financial assistance with these expenses so 
that participants can put their limited resources towards other living and emergency expenses 
(e.g., groceries, car maintenance costs). The Counselor recommends a funding award amount 
and provides it to the Compliance Officer. The officer conducts her own review of the 
participant’s financial need and determines an award amount that takes into consideration 
tuition, books and uniforms, and any need for financial assistance with childcare or other 
expenses. The Compliance Officer identifies funding sources to cover the participant’s time in 
the program. VIDA uses several funding sources for most participants. Staff first apply resources 
from the EDC, city, and/or county in which each participant resides and then identify 
foundation or corporate grant funds that can be applied more flexibly across participants. 
Finally, the Director of Finance reviews and confirms the award amount and proposed funding 
sources, entering the information into VIDA’s accounting system. 


Counseling 

VIDA staff view academic and personal counseling to be an essential component of the program 
model and critical in assisting participants in completion of their certificate or degree. The day- 
to-day function of staff in this role is to assist participants in articulating academic and 
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employment goals, developing an attainable academic plan for full-time enrollment in school, 
and assisting participants with personal and academic needs. To this last point, staff noted that 
among their target population, a family, financial, or health crisis commonly results in the 
participant leaving college. When challenges arise, the Counselor assists in thinking through 
ways to address the issue that will let the participant continue in training. 


VIDA participants attend mandatory weekly counseling sessions led by VIDA’s Counselors. *” 
Three times a month, VIDA holds the sessions in a group setting with other VIDA participants, 
and once a month, the sessions are one-on-one. Staff capacity primarily dictates the proportion 
of group versus individual counseling sessions, as many more Counselors would be needed to 
provide more frequent one-on-one meetings. VIDA Counselors reported, however, that they 
believe there is value in bringing participants together as a group to learn from one another’s 
experiences and build a peer support network they can draw on when academic or personal 
challenges arise. For the convenience of 
participants, Counselors travel to their colleges 
to conduct both types of counseling sessions. 
Counselors monitor participants’ attendance 
closely. Participants sign in at the beginning of 
each group session and Counselors contact 


participants if they do not attend in a given e Time management, balancing school and family, 
and prioritization 


VIDA Group Counseling Session Topics 


e VIDA expectations, participant responsibilities, and 
financial aid overview 


e Budgeting and saving 


e Paying for education 


week. Counselors log individual counseling 
sessions in the program database using case 
notes. The remainder of this section discusses 
the structure of the group and individual 


e Notetaking, study strategies, and overcoming 
procrastination 


e Test preparation, test anxiety 


sessions. 


Group counseling. Within a week of the intake 
appointment, participants begin attending group 
counseling sessions, which typically last 

45 minutes to an hour. Participants attend these 
sessions even if classes for their program of 
study have not yet begun. Each Counselor leads 


Health and wellness 

Conflict resolution in the workplace 

Job interview preparation and skills; resume writing 
Communication and listening skills 

Attitudes toward school and family 


Overcoming challenges 


group sessions with the participants in his/her caseload, often offering multiple sessions per 
week with the same content so that participants can choose one that fits their schedule.*° The 


39 


40 


Following the academic calendar, counseling sessions do not meet over spring break (March), summer break 
(August), or winter break (late December to early January). 


Sometimes two Counselors might combine their group counseling sessions on a given campus either because 


they believe participants would benefit from interacting with a broader group or because not enough 


classroom space is available to hold separate meetings. 
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size of the group ranges from five to 20 participants who are enrolled in a variety of different 
programs at the college. 


Each year, prior to the start of the fall semester, VIDA’s Counselors develop a set of topics for 
the coming year’s group sessions. They select topics to address issues or barriers that 
participants commonly encounter and schedule each to coincide with the point in the academic 
year when the topic is most relevant. For instance, sessions might cover study skills, time 
management, and household budgeting early in the academic year; test-taking skills, stress 
management, and wellness around midterms; and resume development and job search 
strategies near the end of the year. Occasionally, a guest presenter participates, such as a 
member of a college’s career services department who describes its resources. Counselors 
reported that the group session format includes lecture, group discussion, partner or small 
group activities, and individual reflection. 


Individual counseling. Once a month, each participant meets individually with his/her 
Counselor for up to an hour. The appointments typically take place on the participant’s college 
campus, although there is flexibility to meet elsewhere if more convenient for the participant. 
Counselors check in with participants on academic issues (e.g., how classes are going, grades, 
attendance), finances (e.g., changes in a spouse’s income, any issues paying household bills), 
and personal circumstances (e.g., changes in living situation, family matters). Counselors refer 
to a VIDA form to guide the conversation and ensure consistency and continuity across 
meetings. When participants bring up an issue, the Counselor engages them in discussions 
about how to handle it and helps them consider options. For example, one mother’s grades 
were dropping because she had to pick up her young children from school when she otherwise 
would be studying. After talking with her Counselor, she arranged for a family member to pick 
them up, and VIDA arranged financial assistance to compensate the family member for her 
time. 


For certain academic issues, Counselors connect participants to college resources, such as the 
tutoring center or an academic advisor. If a participant begins to fail a class, VIDA requires the 
participant to meet with the instructor and academic advisor and turn in a form signed by them 
as evidence that the meetings occurred. Twice a semester, participants must ask each 
instructor to complete a progress report on their attendance, punctuality, test performance, 
and grades. The Counselor reviews the completed form with each participant to determine 
whether any interventions or support service adjustments are necessary. Participants also must 
periodically submit copies of their grades and transcripts to help Counselors track their progress 
toward degree completion. 


Sometimes issues arise that require Counselors to refer participants to an entity outside the 
college, such as to a licensed clinical therapist for diagnosis and treatment of depression or 
anger management, to credit counseling for assistance with debt, or to legal services if dealing 
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with divorce. Counselors reported that because they see participants weekly and regularly 
monitor their attendance and grades, they are able to identify relatively quickly that a 
participant might be having trouble. In these instances, the Counselor meets with the 
participant outside of their scheduled sessions to address the issue before it escalates. 


Despite efforts by both participants and VIDA, sometimes circumstances arise to keep a 
participant from meeting the requirement of full-time enrollment and other program 
obligations. For example, participants may need to attend to a family emergency or work full- 
time because their financial circumstances change. VIDA attempts to work with these 
participants to keep them engaged in the program in some capacity so they can resume at a 
later point. The program allows participants to be on “active hold” when they are not enrolled 
in training, because they are awaiting admittance to a college program or a personal situation 
prevents them from attending classes. During active hold, they continue to meet regularly with 
the Counselor, who helps them address challenges and encourages their return to full-time 
participation. When they have not participated fully for more than a semester, VIDA places 
such participants on “hold” status, during which they do not meet with their Counselor, but 
VIDA staff contact them periodically to attempt to reengage them. VIDA permits participants to 
be on hold for up to a year before exiting them from the program, According to staff, some of 
these participants eventually return to complete their program of study. 


Financial Assistance 

The type and amount of financial assistance provided for each participant varies according to 
VIDA’s assessment of individual need. VIDA designed financial assistance in this way to 
maximize the use of its own resources while also providing participants with a level of financial 
support commensurate with their needs. Most common types of financial assistance are the 
following: 


Tuition assistance. Tuition varies by institution and program of study. Participants obtain a 
statement of charges from their college that lists tuition and fees and submit it to their 
Counselor. The Counselor then prepares a tuition memo that states the amount of tuition VIDA 
will pay. The Counselor, Compliance Officer, and Director of Finance sign the memo, and it is 
faxed to the college to serve as a purchase order for the college to bill VIDA. VIDA requires that 
participants first access any financial aid programs for which they qualify (e.g., Pell grants, 
federal loans, state grants); VIDA then determines its contribution toward any remaining 
tuition, up to a maximum of $5,500 in a 12-month period. Participants are responsible for any 
remainder. 


Education-related assistance. VIDA covers the cost of course-related books, tools, uniforms, 
and other materials. Participants provide a cost estimate from the college or university 
bookstore for the required materials, and the Counselor and Compliance Officer determine 
which materials VIDA will pay for, which typically is all required materials not paid for by VIDA 
in a previous semester. For participants enrolled in the College Prep Academy, VIDA pays 
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college entrance exam fees up to twice. Participants who must sit for the exam more than twice 
in order to pass pay those additional fees themselves. 


VIDA also will cover childcare and transportation expenses related to attending training: 


e Childcare. VIDA requires that participants first apply for childcare assistance from Texas 
Child Care Management Services (CCMS); if approved, then VIDA covers the co-pay for 
childcare. If denied or put on the CCMS waitlist, participants can seek childcare through 
another provider, and VIDA will assist with the cost upon receipt of a copy of the 
contract that states the cost of care. In both situations, VIDA pays the childcare provider 


directly on behalf of the participant. ; i : 
VIDA’s Community Service Requirement 


e Transportation assistance. Very little VIDA requires participants to contribute to two 
public transportation exists in the Lower community service events per semester. 
Rio Grande Valley, and many participants Underlying this requirement is the idea that because 
travel long distances to get to classes. participants benefit from the contributions of their local 
= ; 6 _ : county or city and grant makers that funded the VIDA 
VIDA takes into account a participant’s program, they should give back to their communities. 
distance from home to school and then In addition, VIDA sees community service as a way to 


foster civic engagement in participants, which aligns 


calculates a gas allowance accordingly. with the organization’s roots in community organizing. 


Participants receive the allowarice at the Among the activities to which VIDA participants have 
end of the month after they provide a contributed are local cleanup events, assisting local 


reco rd of class attenda nce. VIDA does not foodbanks, and representing VIDA at public events as 
.. a way to advertise the program and generate support 
pay for days the participant was absent. for it. If unable to volunteer at an organized event, 


: ; participants can engage in a service activity on their 
Emergency assistance. In rare instances VIDA own, such as by donating blood. 


provides financial assistance for unexpected and 

unbudgeted personal expenses that participants would have difficulty paying, and for which 
they have no other source of assistance, such as for a car repair. VIDA provides such assistance 
as a last resort after its managers, staff, and the participant explore all other options. 


Counselors review participants’ financial needs at the start of each semester. At that time, 
participants provide updated information on their income and expenses and obtain a current 
statement of charges from their college or university. VIDA will make adjustments to the level 
of funding provided to or on behalf of the participant if major changes occur from the previous 
semester. 


3.3.3. Education and Training 


During most of the PACE study period, VIDA participants enrolled in academic and technical 
programs at one of the public institutions of higher education in the region (described in 
Section 3.2.2). VIDA supported participants in various programs of study at each institution. 
This section discusses the program offerings and describes the College Prep Academy. 
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Certificate and Degree Program Offerings 

VIDA commits to funding up to two years of participants’ education, enabling them to complete 
a one-year or longer certificate program or a two-year associate’s degree,” VIDA also enrolls 
participants pursuing their last two years of a four-year bachelor’s degree.” 


VIDA enrolls applicants interested in pursuing education and training in programs of study with 
the potential to lead to jobs in high-demand occupations in the Lower Rio Grande Valley. During 
PACE, VIDA supported programs of study in the nursing, allied health, technology, 
manufacturing, business, education and social services, and specialized trades sectors (see 
Exhibit 3-3). 


VIDA occasionally also supported participants with immediate financial need to enroll in very 
short-term programs (less than one year) that resulted in certificates that VIDA identified as 
being in high demand. The two programs in this category were the four-week Commercial 
Drivers Licensing (CDL) Truck Driving program and the one-semester Patient Care Assistant 
program. 


* Later, in Chapter 4, we distinguish certificates by the number of necessary credits because that is how the 


state of Texas categorizes them. Level 1 requires 15 to 42 credits, and Level 2 requires 43 to 59 credits. For 


comparison, associate’s degrees typically require 60 credits and bachelor’s degrees 120. 


“2 VIDA funds the last two years because its founding leaders learned from the universities that finances became 


difficult for students in their final years of a four-year program; VIDA theorized that tuition and other financial 
assistance provided then could help students complete coursework and attain a degree. 
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Exhibit 3-3. Examples of Programs of Study Supported by VIDA 


PACE 


Area of Study 


Certificate Programs 
Length: 1 Year 


Associate’s Degree Programs 
Length: 2 Years 


Bachelor’s Degree Programs 
Length: 4 Years, 
VIDA Funds Last Two Years 


Nursing 


Licensed Vocational Nursing 


Associate’s Degree in Nursing 


Bachelor's of Science in Nursing 


Allied Health 


e Dental Laboratory Technician 
e Medical Coding Specialist 


Dental Laboratory Technician 
Dental Hygiene 

Diagnostic Medical Sonography 
Emergency Medical Technician 
Health Information Technology 
Medical Laboratory Technology 
Medical Office Management 
Physical Therapy Assistant 
Radiologic Technology 
Respiratory Therapy Care 
Occupational Therapy Assistant 


e 
e 
e 
e 
e 
e 
e 
e 
e 
e 
e 
e Surgical Technician 


None 


Technology 


e Computer Systems Management 
Technology 

Computer Science Software 
Development 

Computer Aided Drafting 
Technician 

Computer Support Specialist 
Digital Media Design Technology 
Multimedia Specialist 


e Architectural Design and 
Engineering Graphics Tech 

e Computer Aided Drafting and 
Design Technology (CADD) 

e Biomedical Equipment Technology 

Chemical Environment 

Technology 

e Computer Science Software 

Development 

Computer Network and Security 

Technology 

Computer Web Development 

Digital Media Design Technology 

Networking Specialist 

Telecommunication Technology 

Aviation Maintenance Technology 


None 


Manufacturing 


Business 


None 


Office Management 
Office Specialist 

Legal Office Specialist 
Legal Assisting 


Precision Manufacturing 
Technology 
Mechatronics Technology 
Accounting 

Business Management 
Technology, 
Import-Export Logistics 
Legal Office Management 
Management 

Marketing 

Paralegal Studies 


None 


None 
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Exhibit 3-3. Examples of Programs of Study Supported by VIDA (continued) 


Bachelor’s Degree Programs 
Certificate Programs Associate’s Degree Programs Length: 4 Years, 
Area of Study Length: 1 Year Length: 2 Years VIDA Funds Last Two Years 
Education and | e Basic Peace Officer e Child Care and Development e Early Childhood Studies 
Social e Basic Fire Fighter e Education (elementary, middle and | ¢ Education Generalist 
Services © Police Officer high school) e Math Grades 4-8 and 8-12 
e Social Work Teacher Certifications 
e Criminal Justice Corrections e Science Grades 4-8 Teacher 
Officer Certification 
e American Sign Language e Biology/Chemistry/Environme 
Interpreter ntal Studies Grades 8-12 
e Law Enforcement Teacher Certification 
e Fire Science Technology 
Specialized e Heating, Ventilation, Air e Heating, Ventilation, Air None 
Trades Conditioning, and Refrigeration Conditioning, and Refrigeration 
Technician Technician 
e Automotive Technology e Automotive Technology 
Technician Technician; 
e Welding e Welding 
e Automotive Technology e Automotive Technology 
e Auto Collision Technician e Auto Collision Technician 
e Commercial and Residential e Building Construction Technician 
Electrician e Wind Energy Technology 
e Combination Welding 
e Electrician Assistant 
Marketable e Commercial Driver's License None None 
Skills Truck Driving (four weeks) 
e Patient Care Assistant (one 
semester) 


SOURCE: Program documents. 


College Prep Academy 

VIDA designed the College Prep Academy for participants who had a high school diploma or 
GED and tenth grade or better academic skills, but who had difficulty passing the college 
entrance exam. The College Prep Academy is an accelerated, full-time 16-week program paid 
for by VIDA. Instruction focuses on math, reading, and writing, as well as test-taking skills. 
Participants can sit for the college entrance exam midway through the Academy if they believe 
they are prepared. If they pass the math, reading, or writing portion of the exam, they can stop 
attending those segments of the College Prep Academy. Upon passing the college entrance 
exam, participants enroll in one of the VIDA-supported programs of study. 


VIDA and South Texas College in McAllen originally partnered to offer the College Prep 
Academy. In anticipation of serving more participants during PACE, VIDA worked with Texas 
State Technical College in Harlingen to implement the Academy there, as well. The two colleges 
are about 45 miles apart, and VIDA was interested in offering the Academy in an additional 
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location within the region to make it available to participants who might otherwise not be able 
to travel to South Texas College. 


As discussed in Chapter 4, VIDA found it difficult to increase enrollment in the College Prep 
Academy despite what staff saw as a great need for accelerated remedial education at this 
level. 


3.3.4 Employment Services 


VIDA discourages participants from working while enrolled in the program because the 
demands of full-time education make it challenging to balance class attendance, schoolwork, 
family, and a job. This is especially true for programs such as Licensed Vocational Nursing and 
some other health practitioner programs because they are particularly rigorous and involve 
clinical hours in addition to class time and homework. Staff reported that some participants in 
non-health related programs work, in order to support their families. When participants want 
or need to work, the Counselor reviews their schedules and degree plans and discusses how 
many hours they want to work and at what type of job, as well as how they would fit work into 
their educational schedule (i.e., attending class and doing homework). If their grades begin to 
slip, the Counselor talks with participants about minimizing personal and household expenses— 
such as giving up cable television and reducing spending on clothing—so they work less or stop 
altogether while enrolled. 


VIDA does not offer extensive employment services and job placement assistance upon 
program completion. Because the program supports education and training only in occupations 
that the EDCs and other partners and funders deem in high demand, the expectation is that 
participants should be able to find and secure a job. While enrolled in the program, VIDA 
attempts to provide participants with the tools needed to be successful in a job search: group 
counseling sessions cover topics such as resume development and interview skills; counselors 
review participants’ resumes and refer them to their campus career services department for 
more in-depth assistance; and VIDA can pay licensing exam fees and transportation for 
participants to take credentialing exams. 


In some occupations, particularly those with internships or clinical placements such as Nursing, 
Medical Office Assistant, and Health IT, some VIDA participants have job offers before they 
finish training. However, even with the careful targeting of training to high-demand jobs, 
graduates do not always move immediately into employment. Nursing and many allied health 
professions require licensing exams prior to employment, and these cannot be taken until after 
graduation. In some fields, VIDA graduates may compete for jobs with individuals with more 
experience in the field. Programs that include internships or clinical hours have helped some 
participants bridge this gap by giving them workplace experience prior to entering the job 
market. 
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Once a participant exits the program, VIDA conducts routine follow-up calls with graduates 
every six months for two years to collect data on employment status, wages, and benefits. It 
analyzes and reports this information to VIDA’s funders and prospective funders to 
demonstrate graduate outcomes and make the case for future funding. In the process of 
conducting these calls, if the Retention Specialist-Placement Coordinator learns that a VIDA 
graduate is unemployed or underemployed, the Coordinator inquires whether he/she is actively 
seeking work. Because job placement is not a component of VIDA’s program, the Coordinator 
offers to review such graduates’ resumes and provide job search tips and resources, but cannot 
offer career counseling or job search assistance. Typically, the Coordinator refers unemployed 
VIDA graduates to Workforce Solutions, the local AJC that provides job search assistance, or to 
the career services departments at the local colleges. The Coordinator follows up monthly to 
check on the graduate’s employment status and suggest additional resources. 
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4. Implementation Study Findings 


When it joined the PACE evaluation, VIDA’s program had operated for about fifteen years and 
in that time had established a consistent model with a core set of services provided to 
participants. Once PACE began, the program operated without modification, aside from 
enhanced recruitment efforts. Although the PACE study can only directly address the period for 
which it collected data, VIDA program staff report that the core program model in this period 
was essentially the same approach as before and after PACE. This chapter first presents 
information on VIDA’s implementation of counseling and financial supports. It then describes 
treatment group members’ experiences in the program and their progression in meeting 
milestones in training. Finally, the chapter concludes by briefly examining effects on treatment 
group members’ receipt of services. 


4.1 Implementation of Counseling and Financial Supports 


Counseling and financial assistance form the core of VIDA’s program services. This section 
draws on information gathered from interviews with VIDA staff, partners, and participants to 
discuss the implementation of these services. It also uses data from VIDA’s accounting database 
to describe participants’ receipt of financial supports during the study’s 24-month follow-up 
period. 


e Participants reported that they benefited significantly from the counseling provided to 
them by VIDA; and PACE team observations, as well as Counselors’ and participants’ 
statements, suggest that treatment group members regarded counseling as mandatory 
and received a substantial dosage of these services. 


It is not possible to quantify attendance at counseling sessions because staff recorded this 
information in case notes that were not available to the research team. However, participants 
discussed in interviews the importance of the VIDA Counselors to their experience in the 
program. Participants said that they felt like the Counselors understood what they were going 
through, provided them with encouragement when training felt difficult or personal challenges 
arose, and offered strategies and guidance. One participant said, “| know mine, she’s really 
good about, ‘If y’all have any problems and you need to vent, call us. We’re here. If we can’t 
help you, we’ll get someone to help you.’ They’ve been a great help.” Another participant said 
of the Counselors, “They were always there. They were a call away, a text away. It was 
ridiculous the amount of attention that they responded.” 


In group counseling sessions, the PACE team observed that participants signed an attendance 
sheet at the start of each session and that VIDA staff consistently mentioned how they kept 
close track of whether participants were meeting the program requirement to attend all weekly 
group sessions. In interviews, multiple participants attested that they had to report for weekly 
counseling sessions, including a monthly individual session. Finally, VIDA staff described how a 
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participant’s failure to regularly attend weekly sessions can result in the program withholding 
tuition assistance for the following semester and discontinuing other forms of training-related 
financial assistance, like childcare and transportation. Thus, although we cannot quantify the 
amount of counseling that VIDA participants received, qualitative information suggests the 
majority of treatment group members received a substantial dose of individual and group 
counseling. 


e Almost all participants (99 percent) received VIDA’s assistance with tuition and related 
expenses of attending and completing training, and VIDA provided an average of almost 
$7,000 in direct financial assistance per participant within 24 months of random 
assignment. 


VIDA commits to providing a substantial amount of financial assistance to support participants 
with tuition and school-related expenses, and as needed to offset other costs associated with 
full-time enrollment in training such as transportation and childcare. As discussed in Section 
3.3.2, VIDA carefully calculates the amount of financial support it will provide each participant, 
taking into consideration his/her household finances and the type of program into which the 
participant is enrolling. VIDA typically also applies several sources of funding—such as from the 
EDCs, cities and counties, and grants—toward a given participant’s expenses. Staff said that this 
so-called braiding, along with VIDA’s highly individualized approach to awarding funding, allows 
VIDA to allocate its resources so as to maximize the number of people it can serve. 


Further, VIDA plans its financial resources to accommodate each participant’s needs for the 
duration of his/her program of study. To do so, VIDA staff project future funding and earmark 
sources and amounts for each participant. Staff said that there are no certainties with future 
funding, particularly because they must make annual appeals to the EDCs, cities, and counties. 
To the extent they can, however, they earmark funds at the participant level to avoid enrolling 
more people than they can afford to serve. Braiding funding sources at the individual level also 
provides a measure of protection so that if VIDA loses expected resources from one funder and 
cannot make up the difference through other means, it can still provide a participant with some 
financial assistance. In these instances, VIDA sometimes has to ask participants to pay the 
difference, and the Counselors work with participants to identify resources or budgeting 
measures to help them remain enrolled. 


Exhibit 4-1 shows the percentage of VIDA participants receiving each type of assistance, the 
average amount of assistance provided, and the share of VIDA’s total spending in each 
category. The data are reported for the 98 percent of treatment group members on whom 
VIDA spent any funds in the 24 months after random assignment. In this report, these 

98 percent are defined as “participants.” 


In the two years after random assignment, for the 98 percent of treatment group members for 
whom VIDA expended any financial resources, VIDA spent an average of $6,808 per participant. 
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As shown in Exhibit 4-1, the largest share of spending—71 percent—was for tuition and other 
fees charged by the colleges and universities, such as building fees. Although a much smaller 
percentage of the total, VIDA also dedicated a substantial share of its resources to books 

(12 percent) and transportation assistance (11 percent). Together, tuition, books, and 
transportation assistance accounted for 93 percent of VIDA’s total financial assistance. 


Of the participants who received financial assistance, almost all (99 percent) received 
assistance with tuition and related expenses of attending and completing training, which 
averaged $4,861 per participant. Eighty-nine (89) percent of participants received assistance 
paying for books, at an average of $910 per participant. Thirty (30) percent of participants had 
help paying for uniforms or interview clothing, because VIDA views these as expenses necessary 
for participation in training and attaining a job. Nursing and many of the allied health programs 
required clinical hours as part of the training, for which participants needed scrubs or other 
uniforms. Less common among the types of financial support related to training was assistance 
for tools (23 percent), school supplies (19 percent), and tutoring (three percent). 


VIDA provided financial support for other expenses that facilitated participants’ ability to attend 
training. The majority of these participants—88 percent—received transportation assistance to 
offset the cost of getting to and from classes. Fifteen (15) percent also received assistance 
paying for childcare (averaging $1,173 per participant). Consistent with its policy, VIDA offered 
financial assistance with emergency expenses only on rare occasions, and during the two-year 
follow-up period, only two participants received such support (averaging $650 per participant). 


Exhibit 4-1. Types and Amounts of Financial Assistance VIDA Provided to Program Participants 


Participants Mean Amount of Share of Total 
Receiving Assistance VIDA Spending 
Type of Financial Assistance Assistance (%) Provided ($) (%) 
Tuition, school, service, and building fees 98.7 4,861 70.6 
Books 88.7 910 11.9 
Transportation assistance 87.7 835 10.8 
Childcare assistance 14.9 1,173 2.6 
Fees for nursing and allied health certification exams 33.6 321 1.6 
Tools 23.2 345 1.2 
Uniforms and interview attire 29.8 115 0.5 
Meals for College Prep Academy 7.0 356 0.4 
Tutoring 2.8 519 0.2 
Rental assistance 0.2 4,875 0.2 
Supplies 19.4 49 0.1 
Emergency living expenses 0.4 649 0.0 
Dormitory housing 0.2 760 0.0 
All types 100.0 6,808 100.0 


SOURCE: VIDA administrative record. Data reported for those treatment group members for whom VIDA reported any expenditures 
by month 24 after random assignment. 
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e VIDA staff reported that the program was a set of obligations and incentives and not just 
a set of services; participant interviews reflected this perspective, too. 


VIDA commits to providing a high degree of financial and personal support, and the participant 
agrees to meet VIDA’s program requirements, including maintaining full-time enrollment in 
training, achieving passing grades, and participating in weekly counseling sessions. VIDA clearly 
articulates these expectations before enrollment into the program, which likely dissuades 
individuals who cannot commit to the program requirements from applying. VIDA also makes 
clear to participants that they have responsibilities to the program, but that the program has an 
investment in them, as well. 


A senior VIDA staff person said that a culture of accountability greatly contributes to VIDA’s 
high persistence rate and that from the beginning, VIDA emphasizes that this program is “not 
an entitlement program, not a ‘gimme’ charity,” and that VIDA will commit to participants’ 
success only if they themselves commit. Similarly, participants underscored the demands of the 
program but acknowledged the benefits, as well. One participant said, “The requirements and 
the paperwork—that’s the only thing. You’ve gotta be running and forth and everything, but for 
the assistance, it’s worth it, ‘cause $1,100, and you get your financial aid and then the books, 
too. It is a big help.” 


Consistent with this participant’s words, VIDA staff articulated the view that financial support 
was more than a service; it was a key incentive for both attracting applicants to and fostering 
retention in the program and in training. One Counselor described the financial assistance as 
the program’s “biggest carrot.” However, Counselors said that financial assistance without 
mandatory counseling would be insufficient to achieve program goals, with one describing the 
latter as their “most successful tool” for encouraging retention and completion. VIDA’s 
commitment of financial assistance for the entirety of a participant’s time in the program—to 
the degree that VIDA can anticipate its future funding resources—gives participants a degree of 
certainty around their ability to afford education and to maintain enrollment when other parts 
of their lives may be less stable (e.g., family matters, their spouse’s employment). 


Meanwhile, VIDA Counselors provide participants with the tools to address academic and 
personal challenges that otherwise might derail their participation in training. Participants said 
in interviews that both the financial assistance and counseling were important to them. One 
participant described the value of both while emphasizing the primacy of tuition support, “1 
think VIDA does an excellent job in providing anything. There hasn’t been anything that I’ve 
lacked. They pay your tuition, and that’s probably the biggest thing. I’m so glad that | don’t have 
to worry about it.” 


4.2 Education and Training Participation Patterns 


Through the substantial financial assistance and counseling support provided to participants, 
VIDA’s goal is to facilitate participation in and completion of certificates, associate’s, and 
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bachelor’s degree programs in high-demand occupations. This section discusses the education 
and training experience of treatment group members, including the completion of key 
milestones. Using VIDA administrative records and financial data, as well as records from the 
local colleges and universities, the analysis reports the overall level of participation and 
completion rates over a 24-month follow-up period. 


e Nearly all treatment group members (98 percent) participated in occupational training, 
and the vast majority (91 percent) completed college credits. About two-thirds 
accumulated 30 or more credits within the follow-up period. 


Exhibit 4-2 (below) shows the proportion of all treatment group members who achieved key 
education and training milestones associated with VIDA’s program. Data are presented by 
educational attainment at “baseline” —when program applicants enrolled in the PACE study 
and were randomly assigned. Almost all treatment group members (98 percent) participated in 
an education or training activity, which includes both basic skills education in the College Prep 
Academy (discussed below) and occupational training. The participation rate varied little based 
on participants’ educational attainment at baseline, with slightly fewer (96 percent) of those 
with a high school diploma or GED participating in any education than those with some college 
experience (99 percent). 


Ninety-one percent of treatment group members obtained at least one college credit, and 

66 percent completed 30 or more college credits (by some standards the equivalent of a year of 
full-time college).*? As would be expected, participants’ ability to earn credits within the 
24-month follow-up period was associated with their education level at program entry: 

95 percent of participants with one year of college or more at baseline completed any credits, 
compared to 81 percent of those with only a high school diploma or GED. Seventy-one percent 
of participants with one year or more of college at baseline earned 30 credits or more. 
Although those with lower levels of education were less likely to earn 30 credits, slightly less 
than three-fifths did so, and 32 percent of participants who began in the College Prep Academy 
earned 30 credits or more (not shown in table). 


“The study identified 30 credits as a milestone because it represents a substantial amount, roughly half of what 


an associate’s degree would typically require. 
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Exhibit 4-2. Participation and Completion for Program Participants, by Educational Attainment at 
Baseline 


High School LessThanOne OneYearof All Treatment 
Diplomaor YearofCollege College or Group 
GED (%) (%) More (%) Members (%) 
Participated in any education or training 96 99 99 98 
Participated in College Prep Academy 28 6 2 9 
Continued to occupational training 23 6 1 7 
Participated in occupational training 92 99 99 97 
Completed any college credits 81 89 95 91 
Completed 30 college credits 57 58 71 66 
Obtained credential or degree 31 40 69 55 
Certificate (Level 1 or 2) 25 22 32 29 
Associate’s or bachelor’s degree 8 17 42 30 
Still attending training at end of follow-up period 50 44 38 42 
Obtained credential 15 15 23 20 
Did not obtain credential 34 30 15 22 
Not attending training at the end of follow-up period 47 54 51 57 
Obtained credential 15 25 47 36 
Did not obtain credential 32 30 14 21 
Sample size 111 81 286 478 


SOURCE: VIDA administrative records and local college records. 

NOTES: “Participated in any education or training” is defined as both the VIDA administrative records indicate enrollment in either 
College Prep Academy or an Occupational Training program, and the participant received any spending from VIDA. “Participated in 
College Prep Academy” and “Participated in occupational training” are based on VIDA administrative records. Completion of credits, 
obtained degree, and participation at follow-up are based on local college records. 


e The College Prep Academy succeeded in preparing participants for entry into college-level 
occupational training programs, although VIDA was not able to increase enrollment in the 
Academy or expand to additional colleges as intended. 


As shown in Exhibit 4-2, nine percent of treatment group members began their participation in 
the College Prep Academy, because they needed instruction in math, reading, and writing in 
order to pass the college entrance exam. These students generally had a high school diploma or 
GED as their highest level of education at baseline. More than a quarter (28 percent) of 
treatment group members with only a high school diploma or GED began in the College Prep 
Academy. Enrollment in the Academy was much less common for participants with some 
college experience, as they were less likely to require developmental education.““ However, 
VIDA Counselors occasionally identified participants during intake who they felt would be more 
successful in occupational training if they first received the remedial instruction provided by the 
Academy. After completing the College Prep Academy, the majority of participants (85 percent) 
enrolled in occupational training (not in exhibit). 


“Since self-report is the basis for “some college,” it’s possible that some individuals reported developmental 


education courses as “some college.” 
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Though the College Prep Academy had historically enrolled a small proportion of VIDA 
participants, during the PACE study, VIDA planned to increase enrollment in the Academy. Its 
analysis of educational attainment levels in the region, where approximately one-fifth to one- 
quarter of residents held a high school diploma or equivalent but had not attended college (see 
Exhibit 3-1), suggested that there was a demand for accelerated remedial education. VIDA 
tailored some of its advertisements to the target population for the College Prep Academy with 
messaging such as “Are you having trouble passing the college entrance exam? VIDA can help.” 
In addition, VIDA worked with Texas State Technical College in order to offer the College Prep 
Academy there. In doing so, it hoped to serve participants for whom it was otherwise difficult 
to attend the Academy when offered only at South Texas College (about 45 miles away). 


However, VIDA was unable to expand the College Prep Academy as planned. First, it was more 
difficult to recruit interested and eligible participants than expected. Program staff reported 
that they had encountered challenges in the past with enrolling enough people to fill a class, 
largely because the Academy requires full-time enrollment for 16 weeks, to which only a select 
group of people can commit (e.g., unemployed, those who work nights or weekends). During 
PACE, however, VIDA had funds to run advertisements in newspapers and on the radio, so it 
expected to reach a new audience who could benefit from the College Prep Academy. 
Ultimately, staff did not know why recruitment was challenging, though they speculated that 
perhaps a difficult economy contributed to people being unwilling to stop working even in low- 
wage occupations in order to enroll. 


Second, VIDA discontinued the College Prep Academy at Texas State Technical College after 
running it three times there. VIDA staff came to believe that the college was not the right 
partner for the Academy since they had difficulty identifying instructors who understood and 
bought into the concept behind the model. In addition, Texas State Technical College proposed 
changes to the Academy curriculum that VIDA did not support, as VIDA’s preference was to 
replicate the curriculum developed by South Texas College. During PACE, the College Prep 
Academy operated twice at South Texas College with cohorts that began in October 2011 and 
June 2014, and three times at Texas State Technical College with cohorts that began in April 
2012, August 2012, and February 2013. 


e More than half of all treatment group members obtained a credential or degree within 
24 months, and it was about equally as common to obtain a certificate as an associate’s 
or bachelor’s degree, 29 percent and 30 percent respectively. 


VIDA achieved high rates of credential and degree attainment. Fifty-five percent of treatment 
group members attained a credential or degree during the 24-month follow-up period (see 
Exhibit 4-2). Twenty-nine percent received a Level 1 or Level 2 certificate, and 30 percent 
received an associate’s or bachelor’s degree; among these, some obtained both a certificate 
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and a degree.” The state of Texas categorizes certificates by the number of necessary credits— 
Level 1 requires 15 to 42 credits, and Level 2 requires 43 to 59 credits.*° As described in 
Chapter 3, VIDA typically supports only programs that require a year or more, supporting those 
shorter of the Level 1 certificates only as exceptions. However, in some longer-term programs 
that VIDA supports, students earn a shorter-term credential partway through the program; for 
example, Patient Care Assistant on the path to Licensed Vocational Nurse (LVN). Fifty-seven 
percent of treatment group members who received a Level 1 certificate accumulated credits 
subsequently within 24 months (not shown), suggesting that for those participants, the 
certificate may be a transitional, but not final, credential. By contrast, 43 percent (or 16 percent 
of the full sample) had not yet earned credits after receiving a Level 1 certificate. 


As would be expected, overall certificate and degree attainment was higher among all 
treatment group members with a year or more of college (69 percent) than for those with less 
than one year of college (40 percent) or a high school diploma or GED (31 percent). Participants 
who entered VIDA’s program with some college credits and experience were further along in 
their training and thus could more readily earn a credential during the 24-month follow-up 
period. Again as would be expected, a similar, but more pronounced, pattern was true for 
associate’s or bachelor’s degrees (42 percent for those with one year or more of college, 

17 percent for those with less than a year of college, and eight percent for those with only a 
high school diploma or GED). 


Among those who obtained certificates, there was less variance by educational attainment at 
baseline, perhaps because certificate programs were shorter term, allowing for participants 
who entered VIDA with a high school diploma or GED to “catch up” with those who started the 
program with some college experience. Almost one-third of participants who entered with one 
year or more of college obtained a certificate, compared with 22 percent among those with less 
than one year of college and 25 percent of those who entered with a high school diploma or 
equivalent. 


e Persistence in occupational training programs was substantial, with slightly more than 
four in 10 treatment group members still attending training at the end of the 24-month 
follow-up period. 


Two years after random assignment and enrollment into VIDA’s program, 42 percent of the 
treatment group had persisted in training (see Exhibit 4-2). Among those still attending, about 
half were still working toward a credential and the other half had obtained a credential, 
suggesting they continued in their educational pathway to attain further certificates or degrees. 


“© As noted in Chapter 3, VIDA funded the last two years of a participant’s bachelor’s education. 


“© For comparison, associate’s degrees typically require 60 credits and bachelor’s degrees 120. 
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Among treatment group members who had only a high school diploma or GED at baseline, half 
were still attending training at the end of the follow-up period (15 percent had attained a 
credential, and 34 percent had not). Among the 47 percent of treatment group members who 
had a high school diploma or GED at baseline and were no longer attending training, fifteen 
percent had obtained a credential, whereas 32 percent had not. Participants who started VIDA 
with at least some college experience fared better: among those with less than one year of 
college experience at baseline, one-quarter had obtained a credential and 30 percent had not; 
and among those with a year or more of college, 47 percent had obtained a credential 
compared and 14 percent had not. 


Even with VIDA’s rich set of services, some participants, about one-fifth, were not able to 
remain enrolled through completion of a credential. The analysis suggests that those with lower 
educational attainment levels at baseline were more likely to stop attending training before 
earning a credential. It is possible that some of these students may be temporarily out of school 
while they address personal issues and may return to VIDA later to resume their studies. 


e Training in nursing and allied health professions, particularly the former, were the most 
commonly attended by treatment group members, followed by programs in education, 
social services, and specialized trades. 


As shown in Exhibit 4-3, three quarters of treatment group members receiving occupational 
training enrolled in nursing or allied health, according to VIDA administrative records. Half of 
those enrolled in health training pursued nursing programs. One-quarter enrolled in Licensed 
Vocational Nurse (LVN) certificate programs, 19 percent enrolled in Associate’s Degree in 
Nursing (ADN) programs, and eight percent in a pathway program that advanced participants 
from LVN to ADN. Other health programs, attended by one-quarter, included certificate and 
associate’s degree programs for Radiology Technician, Patient Care Assistant, Respiratory 
Therapist, Surgical Technician, Health Information Technology, and a Bachelor’s of Science in 
Nursing (BSN). 


Both the high demand for nurses in the region and VIDA’s referral partnerships may produce 
the high level of enrollment in nursing. As discussed in Section 3.2.2, local economic 
development corporations, VIDA staff, and VIDA Board members—some of whom held 
administrative positions in the region’s hospitals—reported a demand for skilled nurses and 
allied health workers, particularly the former, with employers often having to recruit from 
outside the region to fill open positions. One hospital administrator interviewed said that 
participants in ADN and LVN training programs were almost guaranteed jobs upon completion 
and passage of licensing exams because of the vacancies in these positions among hospitals, 
clinics, and physicians’ offices. It is also possible that VIDA’s college partners referred a high 
proportion of nursing students to VIDA for assistance in completing their training programs. 
One college administrator said that the cost of nursing programs (and allied health programs) 
tended to be higher than other types of programs, and the training was sufficiently demanding 
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that participants often found it difficult to work, attend school, and complete required clinical 
hours. This administrator’s institution reportedly referred more nursing and allied health 
students to VIDA than students from other less costly programs. 


Exhibit 4-3. Distribution of Industry and Training Occupations, for Program Participants Who 
Attended Training 


Industry/Occupation Participation (%) 
Nursing and Allied Health 75 
Licensed Vocational Nurse (LVN) 25 
Associate's Degree in Nursing (ADN) 19 
LVN to ADN progressive 8 
Allied health 24 
Education and social services 14 
Specialized trades 11 
Business 9 
Technology 
Marketable skills 3 
Manufacturing 
Sample size 464 


SOURCE: VIDA administrative records (contained in VIDA’s CTK database). The sum of the percents exceeds 100 percent 
because participants could enroll in more than one training program. 

After nursing and allied health, VIDA participants most commonly enrolled in programs in 
education and social services (14 percent), including Social Worker, Criminal Justice Corrections 
Officer, Teacher, and Sign Language Interpreter. Eleven percent of treatment group members 
participated in specialized trades programs such as Heating, Ventilation, Air Conditioning, and 
Refrigeration Technician and Automotive Technician. Nine percent of participants pursued 
business-related programs including Business Management Technology, Office Management, 
and Accounting, whereas seven percent enrolled in technology programs such as Biomedical 
Equipment Technology, Computer and Telecommunications Technology, and Aviation 
Maintenance Technology. A small proportion of treatment group members (three percent) 
participated in a short-term marketable skills program, such as Patient Care Assistant training, 
that VIDA supported to help participants attain a certificate and become employed quickly. Few 
participants (one percent) enrolled in manufacturing programs such as Precision Manufacturing 
Technician and Mechatronics Technician. 


4.3 Impact on Receipt of Services 


This section focuses on the degree to which VIDA affected participants’ receipt of counseling 
and advising and their receipt of employment services. These analyses, which rely on PACE 
follow-up survey data, expand the previous analyses of treatment group experiences based on 
VIDA program data and college records. Using survey data allows the comparison of treatment 
and control group members’ programmatic experiences, to understand how differences might 
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lead to impacts on more distant outcomes. (Exhibit 4-4 below briefly explains how to read the 
impact table here and in Chapter 5.) 


Exhibit 4-4. How to Read Impact Tables 


Impact table Exhibit 4-5, as well as impact table exhibits in Chapter 5, list the outcome measure in the analysis in the left-most 
column (Outcome), with the unit of that outcome in parentheses (e.g., “(%)’). 


The next column (Treatment Group) presents the treatment group’s regression-adjusted mean outcome, followed by the 
control group’s actual mean outcome (Control Group column). The regression adjustments correct for random variation in 
baseline covariates between the two groups (and thus differ slightly from the raw means). The next column (Difference) lists 
the impact—that is, the difference between the treatment and control group means. 


The next column, Standard Error, is a measure of uncertainty in the estimated impact that reflects both chance variation due 
to randomization and any measurement error. 


The final column, p-Value, is the probability that the observed or larger difference between the treatment and control group 
values would occur by chance, even if there was no difference in the outcomes of the two groups. There are several common 
standards for judging statistical significance. In this report, tests are considered statistically significant and highlighted in 
tables (with one or more asterisks) if the p-value is less than or equal to .10. Tests with smaller p-values are separately 
flagged: 


* for .10 


** for .05 
** for 01 


e VIDA had a statistically significant impact on receipt of career counseling and help 
arranging supports for school, work, and family. 


Exhibit 4-5 shows that VIDA had statistically significant impacts on receipt of a number of 
supports. The program produced a statistically significant 14 percentage point difference in the 
proportion of treatment group members who received help from any organization in arranging 
supports to balance school, work, and family compared with control group members 

(31 percent versus 18 percent). VIDA also produced a 13 percentage point impact on treatment 
group members’ receipt of career counseling (36 percent versus 22 percent for control group 
members). 


These effects are likely attributable to the role of VIDA Counselors who assist participants 
throughout the program with issues and decisions related to education, family, and 
employment. Additionally, VIDA Counselors incorporate employability skills such as conflict 
resolution in the workplace in their weekly group sessions. VIDA does not, however, provide 
intensive job search assistance at the conclusion of a participant’s time in the program, and the 
survey showed no effects on job search or job placement. 


Note that the low levels reported by the treatment group for these services are not consistent 
with other more definitive information, including that virtually all VIDA treatment group 
members had an initial assessment, received various forms of financial assistance, and 
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participated in weekly sessions. Most likely respondents interpreted the questions differently 
than how the research team intended. *” 


Exhibit 4-5. Receipt of Varying Supports after Random Assignment 


Treatment Control Impact Standard 
Outcome Group Group (Difference) Error p-Value 
Received assistance from any organization since random assignment (%) 
Career counseling 35.6 22.3 +13.3*** 3.1 <.001 
Help arranging supports for school/work/family 31.2 17.6 +13.6*** 3.0 <.001 
Job search or placement 20.2 22.1 -1.8 2.9 533 
Sample size 434 373 


SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. 
NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between treatment and control groups, are 
summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 


4.4 Summary of Implementation Findings 


Qualitative information suggests that VIDA provided substantial weekly counseling to treatment 
group members and that Counselors and participants regarded weekly counseling as both 
helpful and mandatory. With respect to financial support, VIDA provided an average of 
approximately $6,800 to each participant, with the largest share (71 percent of all such 
expenditures) for tuition, fees, and related school expenses, followed by books (12 percent) and 
transportation (11 percent). These three forms of support were by far the most commonly 
received by participants (88 to 99 percent). Treatment group members also received higher 
levels of assistance in the form of career counseling and help arranging supports for school or 
work. 


Almost all treatment group members enrolled in occupational training, and more than 

90 percent earned at least one credit, with nearly two-thirds earning 30 credits or more within 
a 24-month period after random assignment. Seventy-five percent of participants trained in 
nursing or an allied health occupation. More than half (55 percent) of treatment group 
members received a credential, about evenly split between certificates (Level 1 or Level 2) and 
associate’s or bachelor’s degrees. As expected, students entering VIDA with more education 
were more likely to earn a higher-level credential. More than four in 10 treatment group 
members were still in training at the end of the 24-month follow-up period. Of those 
participants who were not in training at follow-up, 63 percent (36 percent of all treatment 
group members) had earned a credential, and the remainder had not. 


“7 Unlike other PACE reports, we do not report on conditional treatment/control comparisons, because the first 


place of training VIDA participants identified (correctly) was almost universally their college and not VIDA. 
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This chapter reports estimates of VIDA’s early impacts on educational attainment, career 
progress, and non-economic outcomes. The main estimates capture impacts over the 24-month 
period after random assignment, for the full sample of 958 randomly-assigned participants. The 
research team also explored impacts for a longer follow-up period—36 months—for the 
roughly three-quarters of the sample who enrolled before May 31, 2013 and could be observed 
for a longer time in the college records. 


The chapter begins by describing hypothesized impacts and the outcomes the team analyzed. 
Subsequent sections present findings on education, career progress, and non-economic 
outcomes, respectively. In each case, subsections distinguish among confirmatory, secondary, 
and exploratory analyses. 


5.1 Key Hypotheses and Outcomes 


As discussed throughout this report, the VIDA program provides rich financial aid and social 
support for full-time enrollment at local colleges to promote successful attainment of college 
credentials leading to good-paying, high-demand jobs. The program’s theory of change predicts 
their strategies will boost educational attainment by increasing skills (occupation-specific 
abilities and social competencies) and career knowledge; improving financial and social 
resources; and providing better means for coping with life challenges that can interfere with 
school and work. In theory, positive impacts on economic outcomes also should also have 
positive effects on intermediate outcomes—such as career knowledge, work-related skills, self- 
esteem and other psycho-social factors, and life stressors. 


The research team classified outcomes as confirmatory, secondary, or exploratory, according to 
whether they addressed confirmatory, secondary, or exploratory hypotheses about VIDA 
impacts (see Chapter 2). Exhibit 5-1 lists and describes each outcome. 


The confirmatory outcome in the VIDA early analyses is college credits. To support career 
success, the program intends to increase the attainment of one- and two-year credentials and 
degrees as the primary mechanism for promoting career success in the program model. Though 
credential attainment and related employment are the ultimate goals, it seemed likely that 
while some VIDA participants would have earned college credentials in 24 months, many others 
would still be in training. Because it would apply to participants in both situations, the number 
of college credits earned appeared to be the clearest indicator of success for “early impacts.” 
This measure includes credits from academic courses, such as sociology or psychology, and 
credits from technical courses, such as nursing or respiratory therapy. It does not include 
developmental education or college success credits. 
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The team used college records from VIDA’s five partner colleges to estimate program impact on 
credits earned. Analysis of National Student Clearinghouse (NSC) records indicated that 99 and 
98 percent of treatment and control group members attending college after random 
assignment, respectively, attended these colleges, and no other NSC-covered institutions. 


Secondary outcomes include additional education outcomes and a number of indicators of 
early career progress. Tests of hypotheses with respect to these outcomes capture additional 
early effects suggested by VIDA’s theory of change and, as with the confirmatory outcome and 
hypothesis, have an expected direction. As Exhibit 5-1 shows, secondary outcomes include 
several sets of additional measures of educational and career progress such as enrollment full 
time in academic and technical courses, average months of college enrollment, receipt of a 
college credential, employment at or above a specified wage, and employment in a job 
requiring mid-level skills. 


Exploratory outcomes relate to potential additional evidence on program impacts, generally for 
outcomes of interest with some, though less certain, expectation for effects. The research team 
expected college experience, advising, and VIDA material supports to have positive effects on 
assessed career progress, psycho-social skills, and reduced life stressors. 


Also as an exploratory analysis, the chapter estimates impacts separately for subgroups of 
sample members who came to random assignment with varying amounts of college experience, 
and tests for statistically significant difference across these impacts. 
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Exhibit 5-1. Outcomes in the Impact Analysis 
Sample Size 
Treatment Control 
Outcome Description Data Source Group Group 
Confirmatory (Confirmatory Hypothesis) 
College credits earned Average total number of college credits earned College records 478 480 


over a 24-month follow-up period, excluding 
developmental education and college success 
courses 


Secondary (Secondary Hypotheses) 


career-supportive relationships in workforce and 
education settings; response categories range 
from 1=no to 2=yes 


Education 
College credits earned Average cumulative number of college-level College records 478 480 
credits earned by the end of successive six- 
month periods, not including developmental 
education or college success courses 
Enrolled full time in Any full-time enrollment (12+ hours) in each of College records 478 480 
academic or technical four successive six-month periods following 
courses random assignment (1-6 months, 7-12 months, 
13-18 months, 19-24 months, 1-24 months) 
Average months of college Total number of months enrolled full time over College records 478 480 
enrollment the 24-month follow-up period 
Receipt of a college Earned a Level 1 (15 to 42 credits) or Level 2 College records 478 480 
credential (43-59 credits) certificate or technical associate's, 
academic associate's, or bachelor’s degree 
within 24 months 
Enrolled in summer school = Any summer school enrollment within 24 months College records 478 480 
Total hours of occupational Total hours at time of interview PACE short-term 434 373 
training at a non-degree- follow-up survey 
granting institution of higher 
education 
Received a credential Ata college within 24 months, another institution At college: 478 480 
at time of survey, a licensing/certification body at College records 
time of survey, any of the above 
At other location: 434 373 
PACE short-term 
follow-up survey 
Career Progress 
Employment at or abovea —_ Earning $12 or more per hour @ PACE short-term 434 373 
specified wage follow-up survey 
Employment in job requiring Whether employed in a job requiring calibrated 434 373 
mid-level skills set of skills based on federal standards > 
Perceived career progress _ Three-item scale of self-assessed career 434 373 
progress; response categories range from 
1=strongly disagree to 4=strongly agree 
Confidence in career Seven-item scale of self-assessed career 434 373 
knowledge knowledge; response categories range from 
1=strongly disagree to 4=strongly agree 
Access to career supports —Six-item scale counting number of types of 434 373 
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Exhibit 5-1. Outcomes in the Impact Analysis (continued) 
Sample Size 
Treatment Control 
Outcome Description Data Source Group Group 


Exploratory (Exploratory Hypotheses) 
Psycho-Social Skills 


Grit Eight-item scale capturing persistence and PACE short-term 434 373 
determination; response categories range from _ follow-up survey 
1=strongly disagree to 4=strongly agree 


Academic self-confidence Twelve-item scale; response categories range 434 373 
from 1=strongly disagree to 6=strongly agree 

Core self-evaluation Twelve-item scale; response categories range 434 373 
from 1=strongly disagree to 4=strongly agree 

Social belonging in school _Five-item scale capturing sense of belonging; 434 373 


response categories range from 1=strongly 
disagree to 4=strongly agree 


Life Stressors 
Financial hardship Two-item scale capturing financial hardship, PACE short-term 434 373 
reported as either an inability to pay follow-up survey 
rent/mortgage or not enough money to make 
ends meet; response categories are either 0O=no 
or 1=yes 
Life challenges Seven-item scale capturing life challenges that 434 373 
interfere with school, work, or family 
responsibilities; response categories range from 
1=never to 5=very often 


Perceived stress Four-item scale capturing perceived stress; 434 373 
response categories range from 1=never to 
4=very often 

Family structure Became pregnant or had a child within 376 328 
24 months °¢ 


Threshold selected because it was close to the 60" percentile of hourly wages among employed control group members. 

» Skill levels based on the federal O*NETS system with thresholds targeted to PACE program target occupations. Occupational 
categories were coded for PACE by Census Bureau staff from standard open-ended survey items. 

° Among female participants only. 


5.2 Impacts on Educational Attainment 


This section describes impacts of VIDA for key measures of educational progress in the first two 
years of the study for the full study sample (treatment and control groups) and longer-term 
impacts for a subsample of early enrollees with longer follow-up. To highlight the confirmatory 
test’s special role as an indicator of whether early impacts are on track, this section first 
assesses findings on the confirmatory outcome and then examines findings for secondary and 
exploratory outcomes. 


e VIDA significantly increased the total number of college credits earned (the confirmatory 
outcome). 


The increase in credits earned for the treatment group strongly suggests that VIDA’s impacts 
were on the right track at 24 months. As Exhibit 5-2 shows, over a 24-month period, treatment 
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group members earned 33.1 credits compared with 27.5 credits for the control group, an 
increase of 5.6 credits, or roughly two typical courses completed successfully. The 27.5 credits 
for control group members reflects the fact that they also had access to the same college 
courses as treatment group members and in many cases were already enrolled in them. In 
addition, an analysis of the pattern of this effect over time, a secondary outcome, showed 
growing effects on credits earned over each six-month period after random assignment (second 
panel of Exhibit 5-2). 


Although these effects on credits may not immediately appear to be large, these impacts are 
among the largest reported to date from random assignment tests of programs aiming to 
increase college success among low-income individuals. (For more detail, see Section 6.1 and 
Scrivener, et al. 2015). To understand why VIDA’s effects on credits are large, it is useful to 
appreciate that the average positive effect of 5.6 credits is not uniformly spread across all 
treatment group members. Direct evidence for this appears below in Exhibit 5-3 which shows 
that some subgroups of individuals have larger effects. The enrollment patterns in the second 
panel of Exhibit 5-2 suggest an additional possibility. This panel shows that many control group 
members enrolled full time in college without the help of VIDA. Although VIDA support may 
help some of their counterparts in the treatment group to take higher course loads, it seems 
likely that many of them have little or no capacity to enroll at higher levels of credit. If so, this 
further supports the idea that the average positive effect of 5.6 credits results from some 
subset of treatment group members (who we cannot identify directly) experiencing a 
substantially larger gain in credits, while others had a small or no gain. Finally, the average 
number of credits per month of full-time enrollment is 4.8 credits for treatment group 
members and 4.9 credits for control group members. This, along with the fact that VIDA’s 
strategic programmatic goal is to increase full-time enrollment and it seems very unlikely that it 
would actually reduce it, suggests that the observed impact on credits is plausibly accounted for 
by those treatment group members who are in college full time but would have dropped out or 
attended part time had they been in the control group.” 


“8 The results are very similar comparing number of credits attained per any enrollment, part time or full time, 


suggesting that the increase in credits earned by treatment group members results from the relatively small 
proportion of them who would have dropped out of college in the absence of VIDA. 
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Exhibit 5-2. Early Impacts on Education Outcomes (Confirmatory and Secondary Hypotheses) 


Treatment Control Impact Standard _p- 
Outcome Group Group (Difference) Error Value 
Confirmatory Outcome 
Total college credits earned within 24 months of randomization 33.1 27.5 +5.6*** 4.4 <.001 
(average) 
00) aT Of: @ O 
Total college credits earned by end of successive periods after 
randomization 
By month 6 11.2 9.6 +1.6** 0.5 <.001 
By month 12 21.3 18.0 +3,3*** 0.7 <.001 
By month 18 28.2 23.4 +49 1.0 <.001 
By month 24 33.1 27.5 +5.6"** 11 <.001 
Enrolled full time in academic/technical college courses during 
months (%) 
Months 1-6 64.6 55.0 +9,6*** 3.0 <.001 
Months 7-12 51.9 40.8 +11.0*** 3.1 <.001 
Months 13-18 31.3 24.4 +6.9*** 2.9 0.008 
Months 19-24 20.8 16.2 +46" 2.6 0.036 
In any month 1-24 75.7 64.8 +10.9*** 2.8 <.001 
Total months of college enrollment (average): 
Full time 6.89 5.61 +128" 0.36 <.001 
Any 15.38 13.14 +2,24** 0.46 <.001 
Receipt of a college credential (%): 
Level 1 certificate 12.1 10.6 +1.5 2.0 226 
Level 2 certificate 15.9 12.7 +3.2* 2.1 .067 
Any certificate 28.0 23.1 4,9** 2.8 .040 
Associate's degree 26.1 22.9 +3.2 2.7 116 
Bachelor's degree 27 24. +0.6 0.9 267 
Any degree 28.8 25.0 +3.8* 27 .080 
Any credential 53.3 45.0 +8.3*** 3.0 .003 
Enrolled in any summer school within 24 months of random 73.2 56.5 +16.7*** 2.9 <.001 
assignment (%) 
Total hours of occupational training at a college that does not 11.8 14.6 -2.8 8.1 635 
grant degrees 
Received a credential from: (%) 
A college within 24 months 53.3 45.0 +8,3%** 3.0 .003 
Another education or training institution by time of survey 0.6 0.9 -0.2 0.6 646 
A licensing/certification body by time of survey 34.4 35.0 -0.6 3.4 567 
Any of the above 62.2 55.5 +6.7* 3.4 .024 
Sample size @ 478 480 


SOURCE: Abt Associates calculations based on data from VIDA partner college records and the PACE short-term follow-up survey. 
NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between treatment and control groups, are 
summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

* The numbers in this row are the sample sizes for estimates based on college records for the full sample. These are all estimates 
except for the last two panels. In the last two panels (total hours and credentials by place), estimates for activity at a college also 
reflect college records for the full sample, but all other estimates in these two panels (including those for activity at any source) 
reflect the subsample who responded to the PACE follow-up survey, 434 treatment and 373 control group members. 
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e VIDA increased the number of total credits earned by individuals in each of three groups 
defined by their education level at baseline, but the effects were larger for those in the 
two lower-level education groups. 


The number of study participants provides limited statistical power for testing subgroup 
differences. As such, the research team confined subgroup analysis to a single characteristic: 
prior college experience. The team prioritized this characteristic because VIDA recruited 
individuals with a substantial range of prior college education, from those already enrolled in 
college programs to those with no college experience. The PACE baseline survey captured the 
range of prior education in three categories—no prior college, less than one year of prior 
college, and one year or more of prior college—but it did not capture current enrollment status. 
The research team hypothesized that individuals who had completed more college education 
would have demonstrated greater capacity to succeed without VIDA’s services, and hence any 
positive effects of VIDA would be larger for those with less prior college experience. To test this 
hypothesis, the study estimated and compared the impacts of these three groups on the 
primary outcome, receipt of college credits. 


Exhibit 5-3 shows that, at least for the subgroup that had already completed one year or more 
of college, this was the case: the increase in credits earned due to VIDA are about three times 
as large for those in the lower two categories (about nine credits) than in the group with one 
year or more of college at baseline (about 3 credits). The test for difference in impacts is 
significant at the five percent level. It is also notable that although there were these differences 
in the size of impacts, VIDA increased college credits earned for each of the three categories, 
significant at the five percent level or lower. Thus, the program was effective for all three 
subgroups, but more so for the two subgroups with less prior college education. 


Exhibit 5-3. Early Impacts on Credits Earned by Baseline Education Level (Exploratory 
Hypothesis) 


p-Value 
Treatment Control Impact Standard for Effect, p-Value for 
Group Group (Difference) Error (One- Differential 
Outcome (credits) (credits) (credits) (credits) Sided) Effects 
Primary Outcome 
Total college credits earned within .031 
24 months of randomization (average) 
High school or GED 28.5 19.0 +9,5*** 2.5 <.001 
Less than one year of college 35.0 25.9 +9,1%** 3.3 .003 
One year of college or more 35.7 32.6 +3,1* L3 011 
Sample size 478 480 


SOURCE: Abt Associates calculations based on college records. 

NOTES: Statistical significance levels for effects are based on one-tailed t-tests tests of differences between treatment and control 
groups. Statistical significance levels for difference in effects are based on two-tailed t-tests tests of differences in impacts. Both are 
summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 
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e The program also increased rates of full-time college enrollment, enrollment more 
generally, and summer school enrollment, without reducing time spent in other 
institutions. 


VIDA increased full-time enrollment in college, which is central to its theory of change. As 
Exhibit 5-2 (above) showed, VIDA increased full-time enrollment in each six-month increment of 
a 24-month period by between about five and 11 percentage points, and by 11 points over the 
full period. These effects were significant at the five percent level, with most at the one percent 
level or lower. About three-quarters of treatment group members were enrolled full time at 
some point. Viewed in terms of months of enrollment within the 24-month follow-up period, 
VIDA increased full-time enrollment by about 1.3 months and any enrollment by about 

2.2 months (both significant at the one percent level). Because both treatment and control 
group members engaged in very little training at non-degree granting institutions, the increased 
enrollment was not caused by a substitution of VIDA-sponsored courses for other occupational 
training. 


Because VIDA provides financial support, including tuition, for enrollment in summer school 
and most non-VIDA financial programs do not, the research team hypothesized that the 
program would increase summer school attendance. As expected, VIDA increased summer 
enrollment by about 17 percentage points (73 versus 57 percent). 


e VIDA significantly increased the attainment of college credentials. 


VIDA significantly increased the receipt of college credentials by eight percentage points: 

53 percent of treatment group members earned a college credential compared to 45 percent of 
control group members (significant at the one percent level). Effects were largest for 
certificates, about 5 percentage points. Approximately two-thirds of this effect, a little more 
than three percentage points, was due to a significant impact on Level 2 certificates which 
require 43 to 59 credits. There was also a significant positive effect (significant at the 

10 percent level) for academic degrees, about 4 percentage points, and this effect was almost 
entirely due to receipt of associate’s degrees. VIDA did not affect receipt of non-college degrees 
or non-academic licensing/certificates. 


e Educational impacts seen for a subgroup of early enrollees (followed for 36 months in 
college records) suggest that VIDA’s full sample impacts will grow over the next year. 


Availability of an additional year of college data for sample members who had enrolled in VIDA 
by the end of May 2013—somewhat more than three-quarters of the full sample—enables 
analyses of a 36-month follow-up period. These analyses provide a suggestive look into what 
future VIDA impacts might emerge. Such analyses are exploratory because longer-term impacts 
might differ for earlier versus later study participants. 


As shown in Exhibit 5-4 (below), VIDA had positive effects on college credits earned, full-time 
enrollment in college (persistence), and receipt of a college credential (college success) at 
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36 months. Further, the increases all were larger in magnitude than those at 24 months. The 
increase in college credits earned was seven credits over 36 months for the early cohort 
(39.4 credits versus 32.5 credits). 


Full-time enrollment at any point in 36 months for the early cohort was 78 percent compared 
with 65 percent for the control group, an increase of 13 percentage points. Looking at the 
pattern of enrollment over time, although point estimates remain positive, they are smaller and 
non-significant over the last three six-month periods. Notably, by the end of the last of these 
periods (months 31-36), few treatment or control group members were enrolled full time, 13 
and 11 percent respectively. 


Finally, there is a highly significant positive effect on college credentials received (significant at 
the one-tenth of one percent level): 69 percent versus 53 percent, a very large impact of 

16 percentage points. A little less than half of this effect (eight percentage points) is due to 
receipt of a certificate (significant at the five percent level), and a little more than half 

(nine percentage points) is due to receipt of a degree (significant at the one percent level), the 
latter due entirely to a higher level of associate’s degrees. 


The research team anticipates that the 36-month effects are likely to hold for the full sample 
for two reasons. First, the early cohort represents over three-quarters of the full sample, and, 
thus, in the full sample analysis will carry over three times the weight of the remaining roughly 
one quarter of the treatment group. Second, the outcome levels and point estimates for the full 
sample and the early cohort are extremely close in magnitude over the first 24 months, a 
period over which both are observed. 


49 F a ai Ft F ae 
This comparison, and the ones in this section below, is descriptive and, because the analyses are exploratory, 


the research team did not test to determine whether the different impacts are statistically significant. 
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Exhibit 5-4. Longer-Term Impacts (36 Months) on Selected Educational Outcomes for Early 
Enrollees (Exploratory Hypotheses) 
Treatment Control Impact Standard 
Outcome Group Group (Difference) Error p-Value 
Total college credits earned by month after random assignment: 
Month 6 117 9.2 +25" 0.5 <.001 
Month 12 21.6 17.4 +4,2* 0.8 <.001 
Month 18 28.8 23.0 +5. 8 11 <.001 
Month 24 33.6 27.4 +6.2"* 13 <.001 
Month 30 36.8 30.3 +65" 15 <.001 
Month 36 39.4 32.5 +6.9%* 17 <.001 
Enrolled full time in academic/technical college courses during months: (%) 
Months 1-6 65.4 53.8 +11.6** 3.4 <.001 
Months 7-12 49.9 38.6 +113" 3.6 .002 
Months 13-18 33.2 24.7 +8.4** 3.3 011 
Months 19-24 20.6 18.2 +2.4 3.0 417 
Months 25-30 14.4 14.4 +0.0 2.6 992 
Months 31-36 12.7 10.6 +2.1 2.5 401 
In any month 1-36 78.2 65.2 +13.0** 3.2 <.001 
Receipt of a college credential within 36 months of random assignment (%) 
Level 1 certificate 13.9 10.1 +3.9* 2.3 .099 
Level 2 certificate 16.6 13.3 +3.3 2.5 .190 
Any certificate 30.4 22.8 +7,.6** 3.2 .020 
Associate’s degree 41.9 32.3 +9.6* 3.4 .005 
Bachelor's degree 4.1 4.3 -0.2 14 870 
Any degree 45.6 36.4 +9,2*** 3.4 .006 
Any credential 68.8 53.0 +15.8*** 3.2 <.001 
Sample sze® 365 _ 


SOURCE: Abt Associates calculations based on data from college records. 

NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between treatment and control groups, are 
summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

* Sample sizes in this row apply to estimates based on college records for sample members who underwent random assignment by 
June 2013. 


5.3. Impacts on Early Career Progress 


This section presents impact estimates for three measures of career progress. Each captures a 
different aspect of self-assessed progress toward career goals: perceived career progress, 
confidence in career knowledge, and access to career supports. Of these measures, VIDA 
appears to affect only access to career supports but the effect is small, an effect size of .12, and 
significant only at the ten percent level.°° Given that the control group scored close to the 
maximum on these measures, there was not much room for VIDA to improve them. 


°° An effect size is a standardized measure of the size of an effect that is defined as the impact divided by the 


pooled standard deviation of the treatment and control groups. Its purpose in this report is to express ina 
standardized manner the size of impacts that have no natural unit of measurement and to allow for 
comparison of the sizes of effects across scales. 


Abt Associates 5. Early Impacts of the VIDA Program 1 pg. 66 


Valley Initiative for Development and Advancement 
Implementation and Early Impact Report 


PACE 
Exhibit 5-5. Early Impacts on Selected Career Outcomes (Secondary Hypotheses) 
Treatment Control Impact Standard 
Outcome Group Group (Difference) Error Effect Size _ p-Value 

Indices of self-assessed career development (average) 

Perceived career progress @ 3.71 3.69 +0.02 0.04 +0.03 652 
Confidence in career knowledge > 3.61 3.59 +0.03 0.03 +0.06 416 
Access to career supports ¢ 1.83 1.80 +0.03* 0.02 +0.12* .090 
Sample size 4 434 373 


SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. 

NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between treatment and control groups, are 
summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

* Three-item scale tapping self-assessed career progress; response categories range from 1=strongly disagree to 4=strongly agree. 
 Seven-item scale tapping self-assessed career knowledge, response categories range from 1=strongly disagree to 4=strongly 
agree. 

° Six-item scale tapping self-assessed access to career supports, response categories range from 1='‘no’ to 2="'yes.’ 

‘ Sample sizes in this row apply to sample members responding to the PACE follow-up survey. 


5.4 Impacts on Other Outcomes 


This section describes other, more distant, outcomes that access to VIDA may affect. Two 
indicators describe employment outcomes: working in a job that pays at least $12 per hour and 
working in a job requiring at least mid-level skills. Though many study participants are still in 
training, others have completed it and may have found jobs. As seen in Exhibit 5-6, top panel 
(below), results show no evidence of impacts for either of the employment outcomes. It is 
uncertain whether positive impacts on employment and earnings will emerge in the longer 
term, although the positive effects on college credentials suggest that they could. That said, 
higher levels of full-time college enrollment tend to initially reduce employment, making it less 
likely that early impacts on employment would be observed in this initial 24-month analysis 
period. 


Positive impacts on educational attainment and self-assessed career prospects create some 
possibility for positive effects on psycho-social skills associated with college success. Social 
scientists have verified the measures of psycho-social skills used in the PACE follow-up survey in 
research settings, but psychometricians have recently raised concern about their use in 
program evaluations. Specifically, individuals in a program that emphasizes these skills may 
come to have higher expectations of their performance than do control group members, and 
thus the treatment group members rate the same level of performance more negatively than 
do control group members (Duckworth and Yeager 2015). This potential for measurement 
biases injected some uncertainty about the direction of expected effects, such that the PACE 
study treats these analyses as exploratory (i.e., subject to two-sided tests). 


For VIDA, results show no evidence of impact (Exhibit 5-6, second panel) for any of four indices 
of psycho-social skills tested. On its face, the program’s enhanced counseling and supports 
might predict positive psycho-social impacts. Successes in training also might improve self- 
assessments of personal qualities and capacities. On the other hand, the program’s emphasis 
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on psycho-social skills was usually not explicit. Finally, there is the potential for measurement 
error, as described above. 


The third group of exploratory outcomes included self-reported indices of life stressors, defined 
as reports of financial hardship, life challenges, and perceived stress. Although VIDA supports 
might help alleviate some pressures, members of the treatment group may have experienced 
higher levels of stress from striving to balance full-time school with other responsibilities. 
Because of these potentially opposing influences, the research team classified hypothesized 
effects on stressors as exploratory in the early analysis. Results indicate no significant effects on 
several measures of stress (Exhibit 5-6, third panel). 


The fourth and final group of exploratory outcomes concerned family structure: whether the 
study participant was living with a spouse and whether the participant had had a child since 
random assignment or was pregnant at the time of the follow-up survey. These outcomes were 
included in the analyses because the literature suggests that education and career progress 
can, in the short run, raise the opportunity costs of marriage and childbearing (Buckles 2008; 
Brand and Davis 2011). The bottom panel of Exhibit 5-6 shows that VIDA’s program produced 
large effects on childbearing among female survey respondents, with a reduction of 

10 percentage points from a base level of 23 percent (a 46 percent reduction)** in women 
becoming pregnant or having a child within approximately 20 months of random assignment 
(the average length of time to survey completion). The finding suggests that treatment group 
members may have seen the opportunity to participate in VIDA as a substantial one that they 
did not want to forego. 


°t Calculation is 10.3/22.6 = .456. 
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Exhibit 5-6. Early Impacts on Other Outcomes (Exploratory Hypotheses) 
Treatment Control Impact Standard _ Effect p- 
Outcome Group Group _ (Difference) Error Size Value 
Indicators of Career Pathways Employment (%) 
Working in a job paying at least $12 per hour 26.5 24.3 +2.2 3.0 +0.05 470 
Working in a job requiring at least mid-level 29.9 28.0 +1.9 3.2 +0.04 554 
skills @ 
Indices of Psycho-Social Skills (average) 
Grit 3.32 3.32 0.00 0.03 0.00 975 
Academic self-confidence ¢ 5.16 5.12 +0.04 0.05 +0.07 345 
Core self-evaluation 4 3.51 3.53 -0.02 0.03 -0.04 .600 
Social belonging in school @ 3.51 3.55 -0.04 0.03 -0.08 272 
Indices of Life Stressors (average) 
Financial hardship & 0.51 0.51 -0.01 0.03 -0.02 830 
Life challenges ' 1.55 1.56 0.00 0.03 -0.01 939 
Perceived stress 9 1.97 1.95 +0.02 0.05 +0.02 140 
Family structure (%) 
Living with spouse 38.1 40.3 -2.1 2.7 -0.06 432 
Had child since random assignment / currently 12.3 22.6 -10.3** 3.0 -0.29"* 001 
pregnant" 


SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. 

NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between research groups, are summarized as 
follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

* Five-item scale capturing sense of belonging, response categories range from 1=strongly disagree to 4=strongly agree. 

i Eight-item scale capturing persistence and determination, response categories range from 1=strongly disagree to 4=strongly 
agree. 

°Twelve-item scale capturing academic self-confidence, response categories range from 1=strongly disagree to 6=strongly agree. 
* Twelve-item scale capturing core self-evaluation, response categories range from 1=strongly disagree to 4=strongly agree. 

° One-item scale capturing financial hardship, reported as inability to pay rent/mortgage or not enough money to make ends meet, 
response categories range from O=no to 1=yes. 

‘ Seven-item scale capturing life challenges that interfere with school, work, or family responsibilities, response categories range 
from 1=never to 5=very often. 

9 Four-item scale capturing perceived stress, response categories range from 1=almost never to 4=very often. 

" Sample sizes for this row are 308 women in the treatment group and 275 in the control group who responded to the PACE follow- 
up survey. 

Sample sizes in this row apply to sample members responding to the PACE follow-up survey. 


5.5 Summary of Impact Findings 


By 24 months after random assignment, VIDA had substantial and growing effects on college 
credits earned. These effects were present for treatment group members who entered the 
program with different levels of education, but were particularly large for those with less than 
one year of prior education. In addition, VIDA positively affected a broad array of educational 
outcomes, including full-time college enrollment, enrollment more generally, and summer 
school enrollment, as well as receipt of college credentials. Finally, for the more than three- 
quarters of the sample with 36 months of follow-up, effects on credits and credentials 
continued to increase. VIDA had few effects on other outcomes. The major exception to this 
was that pregnancy and childbearing were lower for those with access to the program. 
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6. OTe) ale (Uk}(e) abs) 


This concluding chapter summarizes and reflects on the importance of the report’s key impact 
findings. First, to put the magnitude of the impacts in context, the chapter compares results 
with those of another prominent program with similar aspects. Second, the chapter discusses 
possible explanations for VIDA’s positive effects on educational progress and achievement. A 
third section notes some of the broader implications for policy and practices. A final section 
describes the primary questions that longer-term follow-up will answer. 


6.1 How Large Are VIDA’s Effects? 


VIDA’s educational effects are among the largest identified by random assignment studies of 
programs to promote successful completion of postsecondary education and training. A helpful 
way to gain perspective is by comparing the VIDA findings with results from a recent 
experimental evaluation of the City University of New York (CUNY) Accelerated Study in 
Associates Program (ASAP).°? ASAP provides a good comparison because its effects have been 
widely hailed as some of the most promising to date and because the two interventions have 
some generally similar components, such as a requirement to enroll in college full-time, 
intensive counseling and advising services provided by a dedicated staff member, and funds 
provided for tuition, course materials, and transportation assistance.°° 


With respect to credits earned over a 24-month period, the VIDA and the CUNY ASAP programs 
produced almost identical results, an increase of six credits, with almost identical levels of 

33 credits for treatment group members versus 27 credits for control group members.” With 
respect to credentials, CUNY ASAP increased receipt of associate’s degrees by 18 percentage 
points at the end of 36 months of follow-up, whereas VIDA increased receipt of credentials 
(including the full range of credentials and degrees it supports) by 16 percentage points for the 
more than three-quarters of the sample with 36 months of follow-up.” Thus, VIDA’s effects are 


2 As described in Chapter One, the Project QUEST evaluation, although it studies the model which underlies 


VIDA, is not very useful for understanding the size of VIDA’s effects, because the follow-up period for the 
evaluation, six years, is so much longer than current follow-up for the PACE evaluation of VIDA. However, as 
longer follow-up becomes available for the latter evaluation, the former one will become much more relevant 
to understanding the evaluation of VIDA. 


MDRC, the evaluator of CUNY ASAP, describes its findings as “the most positive MDRC has found in over a 
decade of research in higher education” (Scrivener et al. 2015). Another random assignment study with large 
effects on educational enrollment and persistence is the H&R Block College Financial Aid Application 
Assistance, which has not measured credit or credential attainment at this point (Bettinger et al. 2012). 


53 


°*  VIDA’s actual value for the control group is 27.5, but here the value is rounded down in a slight exaggeration 


for simplicity. 
°° Effects on health certificates by Project QUEST were 26 percentage points, but this was after six years and was 
primarily for certificates. Project QUEST actually reduced receipt of Associate’s degrees by eight percentage 


points, as they were not a target of the program. 


Abt Associates 6. Conclusions 1 pg. 70 


Valley Initiative for Development and Advancement 
Implementation and Early Impact Report 
PACE 


strikingly similar to those of a program that has produced very large effects in the context of 
random assignment studies. 


It is important to note that the difference in credentials earned in the two studies may have 
important consequences for their potential employment effects. CUNY ASAP’s effects are just 
for associate’s degrees, and while some of VIDA’s supported credentials are for associate’s and 
bachelor’s degrees, almost half of the total effect for VIDA is for occupational certificates that 
are not degrees. Research suggests that, other things equal, credentials requiring longer study 
have greater economic value than those requiring shorter time (Belfield and Bailey, 2011; 
Jepson, Troske, and Coomes, 2014). However, factors such as gender and the occupational area 
of study are importantly related to exceptions to the general rule. For example, long-term 
nursing certificates can have greater economic payoff for women then associate’s degrees 
outside health (Dadgar and Trimble 2015). Longer-term follow-up for both studies will help to 
clarify the economic payoff of each program. 


6.2 What Explains VIDA’s Large Effects? 


VIDA’s theory of change and findings from the implementation and early impact studies suggest 
how the program might be achieving its positive effects. Because the PACE evaluation design 
provides reliable estimates of the effects of the overall program, but not its particular features 
separately, this section is more speculative than the impact estimates in Chapter 5 and cannot 
definitively explain VIDA’s large effects. 


As discussed in preceding chapters and shown in the theory of change (Exhibit 2-1), assessment 
is a key component of VIDA’s program. First, VIDA screens carefully for individuals with the 
capacity and the motivational commitment to attend school full time. That 65 percent of the 
control group, without access to VIDA’s services, has enrolled full time at some point in a 
24-month period supports the proposition that VIDA staff carefully screen individuals in this 
regard. Second, during the intake process, VIDA staff assess each participant’s individual 
financial need. In this regard, VIDA attempts to determine: Is its financial assistance likely to 
make a difference in the participant’s ability to attend school? To what degree do applicants 
have sufficient financial resources, or access to resources (e.g., family members who can assist 
them), for their non-training expenses that would minimize their need to work while enrolled in 
school? Consistent with its goal of identifying a program population with financial need, VIDA’s 
staff reported that often its partner colleges refer potential applicants to VIDA when they learn 
that their students are at risk of dropping out of school because of difficulty paying for tuition 
and school-related expenses. 


The theory of change also points to support services as factors that may influence program 
impacts. In particular, substantial financial assistance coupled with mandatory participation in 
counseling and compliance with VIDA’s monitoring activities appear to be critical components 
of VIDA’s ability to achieve desired outcomes. With respect to financial assistance, as described 
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in Chapter 4, VIDA participants interviewed in depth nearly universally affirmed the high value 
they placed on the program’s financial support and how it represented a unique opportunity. 
As one participant put it, “I just think it’s a very, very, very, very, very great opportunity that 
VIDA is offering.” In addition, participants often described financial assistance as their main 
motivation for their being in the program: “The reason | got into it, because there was all this 
help of financial—like financial help. If | can get help with that, it’s gonna be very greatly 
appreciated.” And in fact, the program delivered very substantial financial assistance, an 
average of almost $7,000 per participant, primarily in tuition and related fees, over two years. 


VIDA staff recognize the value of the program’s rich and substantial financial assistance to its 
participants and that this gives the organization leverage to enforce its program requirements. 
One staff person described financial assistance as the program’s “biggest carrot,” following that 
with a description of how it can be withheld or reduced if participants don’t comply with 
requirements. Perhaps most centrally, participants have a powerful incentive to cooperate with 
the program’s intensive case management, primarily attendance at weekly counseling sessions, 
which VIDA staff described as their “most successful tool” for encouraging retention and 
completion of college programs. In interviews, participants nearly universally expressed it as a 
given that they attended meetings weekly, with some explicitly calling it out as mandatory. 
Thus, VIDA’s high support appears to be a lever for high expectations. 


Participants frequently described the weekly counseling sessions as highly valuable and 
emphasized the importance of VIDA’s Counselors to their positive experience in the program 
and ability to balance school and personal commitments. One participant described the 
sessions this way: 


When we have our weeklies, they teach us different things. They teach us how to 
take tests, how to study, how to doa resume, what things we have available, 
what resources we have available. It’s just different topics. Then when you met 
one-on-one, it’s to see how good you’re doing and to keep up and keep you 
focused. 


Although the study cannot empirically separate the effects of counseling from those of financial 
assistance, the words of both program staff and participants strongly suggest that the two work 
together synergistically. Participants comply with VIDA requirements in part because of the 
generosity of its financial assistance, and its intensive counseling services help participants to 
remain in school and complete their programs. Thus, high expectations and high support 
appear to go hand-in-hand to improve outcomes. 


From the perspective of the career pathways framework, the VIDA model represents a 
relatively intensive and long-term program concentrating on supports targeting a strongly- 
motivated population with relatively high educational levels. That framework posits that low- 
income individuals typically face multiple barriers to successful completion of postsecondary 
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education, and that failure to address a sufficient number will not substantially improve 
outcomes. VIDA attempts to deal with numerous potential barriers, including lack of sufficient 
financial, social, and personal supports. In addition, it attempts to strengthen skills in critical 
areas, including those related to college success, household management, time management, 
and balancing competing responsibilities. Except for the College Prep Academy, VIDA does not 
introduce new instructional approaches, but rather focuses on providing the supports the 
program believes necessary to sustain participants in college coursework. Thus, it views the 
richness of its services as a comprehensive complement to existing occupational training. 


6.3. What Are Implications for Program Operators and Funders? 


The PACE impact results show that VIDA had substantial positive effects on treatment group 
members’ educational outcomes. Although important questions remain (discussed in the next 
section), the findings in this report suggest several uses that program operators and funders 
may want to make of the results. 


One viable option for program operators and funders is to replicate VIDA’s model in other 
locations. There is proof that such replication is feasible in the fact that the underlying model 
already operates in nine other locations and has done so in several of them for more than two 
decades. Furthermore, the underlying model for VIDA is well-documented in program 
materials, including its administrative and operational procedures. While these features do not 
guarantee that other programs adopting the model will produce the same effects as VIDA, they 
do indicate the model’s strong potential for replication. 


A second viable option for program operators and funders is to consider how they might 
experiment with the model to modify it for other populations and circumstances. The VIDA 
early impact findings, along with those of CUNY ASAP, suggest that the combination of required 
full-time enrollment, intensive counseling, and financial assistance appears to effectively 
increase the number of credits earned and credentials attained. However these three broad 
program components vary substantially in their specific features. As described earlier, CUNY 
ASAP targets associate’s degrees (and transfers to four-year schools), whereas VIDA targets a 
broader range of credentials. The populations of the programs are also quite different in 
important respects: CUNY ASAP’s participants are younger, more uniformly require 
developmental education, and are primarily at the beginning of their college experience. By 
contrast, VIDA’s population is older, only about one in ten has developmental education needs, 
and the large majority already have college experience. Finally, the programs differ in another 
key respect: CUNY ASAP, unlike VIDA, does not restrict its support to exclusively occupational 
programs. 


These variations suggest that, even while building a program based on the three core 
components, there is the potential for even greater programmatic flexibility beyond what is 
observed in the two programs. For example, as described earlier, VIDA is a highly selective 
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program: 65 percent of VIDA’s control group enrolls in school full time at some point within 

24 months without any of its supports, implying a population with both relatively high academic 
preparation and considerable motivation. Given that not all students can attend a full-time 
program, even with VIDA’s level of support, program operators may want to test whether 
providing similar supports to part-time students also might be effective. Similarly, VIDA is 
selective in that more than 90 percent of its participants already had college-level skills and 
some prior college education, and even entrants to the College Prep Academy need to have at 
least tenth grade basic skills. Program operators might want to test whether additional 
instructional supports might make it possible for individuals with lower skills to advance into a 
program based on VIDA’s core model. 


It would be important to evaluate efforts to further transplant or modify the VIDA model, as the 
effects of doing so are unknown, although the Project QUEST evaluation is very encouraging in 
this regard. However, the substantial variation in details between VIDA and CUNY ASAP suggest 
that it could be worthwhile to explore whether further variation, extending the populations 
served, undermines the integrity of the program. 


6.4 What Lies Ahead for the Evaluation of VIDA? 


This report’s findings on VIDA’s impacts on college outcomes over two years of follow-up 
represent some of the largest effects to be established in random assignment evaluations of 
promising postsecondary programs. But, many important questions remain that subsequent 
reports on effects on intermediate outcomes (with 36 to 42 months of follow-up) and long- 
term outcomes (with 72 months of follow-up) will address. These include: 


e Will VIDA’s educational effects remain stable, grow, or shrink? It is encouraging that 
effects are still growing among the portion of the sample with 36 months of follow-up 
(early enrollees). These positive effects might represent permanent improvements over 
the control group, or members of the later-enrolling group could also recover from 
temporary setbacks and reduce differences with treatment group outcomes. 


e Will VIDA’s positive educational effects translate into positive impacts on employment 
outcomes in the longer term, consistent with the program’s theory of change? This 
report examines only a very few employment outcomes. The next report will look at a 
much broader array of employment outcomes, including earnings, hourly wage rates, 
fringe benefits, and stability of employment. 


e Does VIDA have other positive effects on treatment group members and their 
families? Key outcomes to address this question include individual and household 
income, material well-being, and perceived stress. 


e Is VIDA cost beneficial? The research team will examine this question at both 36 and 
72 months after random assignment. Given that VIDA requires substantial up-front 
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investments in both participant time and program resources, the initial answer will be 
quite tentative and the later more definitive. 


Given the strong early results presented in this report, these later reports also will be important 
additions to the career pathways literature. 
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